Using AWS in China

Categories: aws

TL;DR;

A collection of differences between non-China AWS and China AWS and how I’ve solved them.

Recently I got my hands on a project which is to be deployed across multiple regions in the world, one of them being China. Naive as I where, I thought my CloudFormation stacks would just work in China, since they worked so perfectly in both Europe (eu-west-1) and the US (us-east-1). But… I was wrong. Very wrong.

After quite a lot of trial and error, I finally managed to get my modified stack running in China, success!

So, to unburden the pain from you, I’ve compiled a list of all the caveats I’ve found so far.

This blog post assumes you’re using some like CloudFormation, Terraform or running commands via aws-cli. The experience will be different in the AWS Console since it guides you to do the correct choices.

Instance types

This one is quite obvious, but still a bit of a pain.

China is lagging behind a bit when it comes to instance types, so please take an extra look into your scripts before running them to ensure that the instance types you’ve selected are available.

You can find the China instance types at https://www.amazonaws.cn/en/ec2/details/.

AMIs

This really shouldn’t be a surprise at all, since it’s different in all regions. What caught my attention, was that some images (Ubuntu 16.04) aren’t up to date to the current version, it even differs across the two regions in China.

Example: In cn-north-1 Ubuntu 16.04 20180109 is available, but in cn-northwest-1 it’s not. This is a bit weird, since 20180109 is available everywhere, but cn-northwest-1.

I’ve written a small Python script to find all AMIs for Ubuntu 16.04 called image-map.py and it contains this:

#!/usr/bin/env python2.7

import boto3

def main():
	client = boto3.client("ec2")

	for region in regions():
		for image in images(region):
			print "%s: %s" % (region, image.get("ImageId"))

def regions():
	client = boto3.client("ec2")

	for region in client.describe_regions().get("Regions", []):
		yield region.get("RegionName", "")

def images(region_name):
	client = boto3.client("ec2", region_name=region_name)

	for image in client.describe_images(Filters = [ { "Name": "name", "Values": [ "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180109" ] } ]).get("Images", []):
		yield image

if __name__ == "__main__":
	main()

Replace ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180109 with a string that matches your wanted AMI and then run the script twice, one with a AWS “the world” account and one with a AWS China account. Example:

$ AWS_PROFILE=standard_aws_account ./image-map.py
$ AWS_PROFLE=aws_china_account ./image-map.py

This assumes you have setup two AWS profiles. Read more on https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html.

IAM trust entities

Another thing that surprised me is that IAM trust entities sometimes are named differently. The surprise stems from that I assumed the name of the entities to be something that’s internal to AWS, in which case they are free to choose whatever name they want, but this wasn’t the case.

The entities I use are monitoring.rds.amazonaws.com, ecs.amazonaws.com and ec2.amazonaws.com. Both rds and ecs are named the same in China as they are elsewhere, but ec2 is not. It’s called ec2.amazonaws.com.cn in China.

What I did in my CloudFormation template was to create a condition and use Fn::If to select which entity name to use. Example:

Conditions:
  China: !Or
    - !Equals
      - cn-north-1
      - !Ref 'AWS::Region'
    - !Equals
      - cn-northwest-1
      - !Ref 'AWS::Region'
Resources:
  IamRole:
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - !If
                  - China
                  - ec2.amazonaws.com.cn
                  - ec2.amazonaws.com
      ManagedPolicyArns:
        - !If
          - China
          - arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
          - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
      Path: /
    Type: AWS::IAM::Role

This snippet uses the same condition to select the correct policy ARN, since they too are different in China.

There can be other entities that follow the same scheme, but I only came across these since I mainly use RDS, ECS and EC2, but be aware of Lambda and other services that heavily rely on IAM roles/profiles.

ARNs

As I just mentioned, ARNs are different in China as well, but they are quite easy to manage, simple replace arn:aws: with arn:aws-cn: and you’re done!

Given we use the condition China which I defined in the snippet above, we can select the correct ARN doing this:

- !If
  - China
  - arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
  - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

Cross-region VPC peering connections

No big surprise here, given that cross-region capability is quite new and not introduced to that many regions yet, but it still made me a bit sad.

Non-mirrored package repositories

This caused the most pain of all issues. Package repositories aren’t mirrored in China. The Ubuntu repositories are mirrored, so if you only install packages from those repositories, you’re fine. But if you like me, run Docker or Gitlab, you’re gonna have intermittent difficulties reaching the official mirrors.

There is a solution though, since companies inside China offers mirrors that works.

So, for Docker, replace the official deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable with a Chinese mirror provided by Alibaba deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu xenial stable.

The GPG key can be fetched from http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg.

Same procedure is needed for Gitlab, replace the official deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ xenial main with deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu xenial main provided by https://mirror.tuna.tsinghua.edu.cn/ (I guess this is some kind of school). The official key can be used for this repository.

I haven’t found any need for other mirrors yet, but I’m sure there is a need depending on which software you use.

No Route53

Again, no surprise, Route53 isn’t available in China. But, you can configure Route53 with an AWS “world” account and use Route53 in China, since there are edge nodes in China. You just can’t access the API in China via amazonaws.cn.

IAM isn’t global

IAM is supposed to be global, which it almost is, except China. China has their own installation of IAM, so you need to duplicate your users from AWS into AWS China. Not a big issue, but something to be aware of.

Final words

I think it’s amazing that we can use AWS in China, even with the limitations and caveats. I think there’s still a lot for me to learn about China, so I’ll probably revisit this post at a later time and update it with new problems and solutions.

Install Gitlab on Ubuntu 16.04

Categories: ubuntu

TL;DR;

A short description on how to install Gitlab Community Edition on Ubuntu 16.04 without using magic scripts.

If you, like me, dislike using magic scripts to install stuff, then this is the post for you. If not, please read the official installation instructions on https://about.gitlab.com/installation/#ubuntu.

Add Gitlabs APT key

First we need to add the GPG key. This key is used for both the official package repository and the China mirror.

curl -L https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey | sudo apt-key add -

Add the Ubuntu 16.04 APT repository

To add the official repository to APT, run:

echo "deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ xenial main" | sudo tee /etc/apt/sources.list.d/gitlab.list

Bonus: APT mirror in China

If your server is located in China, there’s a great change it might not be able to download Gitlab via the official mirrors. But, don’t fear, there are mirrors inside China.

Run the following command, instead of the previous command to add the China mirror:

echo "deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu xenial main" | sudo tee /etc/apt/sources.list.d/gitlab.list

No matter if you added the official repository or the China mirror, run the following command to update APT:

sudo apt-get update -y

Install required packages

To install all packages needed by Gitlab, please run:

sudo apt-get install -y openssh-server ca-certificates gitlab-ce

The gitlab-ce package will install redis, nginx and postgresql as needed by Gitlab. There is a package called gitlab which doesn’t include these dependencies, but do tread carefully.

Configure and start all services

To setup Gitlab, run the following command:

sudo gitlab-ctl reconfigure

This will fire up Chef and run the included recipes for Gitlab, when done, it will start all services needed for normal operation.

That’s it! Gitlab Community Edition and it’s dependencies are install. Visit http://your_server_ip/ in your webrowser to setup the root password.

If you want to modify the installation (Perhaps to add HTTPS), please consult the official documentation on https://docs.gitlab.com/omnibus/README.html

Holding packages in Ubuntu

Categories: ubuntu

TL;DR;

A quick tip on how to hold packages at a given version in Ubuntu. Will work for other Debian based operating systems as well.

First, install the package at the given version. Here’s how to install Elasticsearch 2.4.1:

sudo apt-get install -y elasticsearch=2.4.1

Verify it’s installed with the correct version by running:

dpkg -l | grep elasticsearch

This should return a line that looks like this:

hi  elasticsearch                    2.4.1                               all          Elasticsearch is a ...

Now it’s time to tell dpkg to hold this version:

echo "elasticsearch hold" | sudo dpkg --set-selections

Verify by running:

dpkg --get-selections | grep elasticsearch

That’s it! From now on, each sudo apt-get upgrade or equivalent won’t upgrade the Elasticsearch package. Neat!

Working at a remote-hostile company

Categories: life

I want to start by saying that I’m quite happy with my current employer, this post isn’t written to speak ill of my employer, it’s just my view on working as a remote employee at a company where the culture isn’t exactly remote friendly.

This post is going to be about the problems you can encounter as a remote employee working for a company where most or all other employees are working at the same office. I’ll try to suggest solutions to these problems, but not all are tested, so implement with causion.

Meetings

While chatting and emailing are great tools, sometimes it’s better to be able to talk to eachother and see eachother. That’s when you need to hold a meeting. The best way to do this is to meet face to face, but this is quite impractical if people are scattered across the globe.

Here’s my list of things to do before, during and after having a meeting:

  1. Write an agenda. Decide how much time to allocate for the meeting. Meetings without an agenda and an end time are wasted meetings.
  2. Book a time when everyone needed for the meeting are able to attend. Put it into a shared with an alert the day before and an hour before.
  3. Decide the conference tool to use and preferably test it before starting the meeting. This can be tested between different attendees or a few minutes before the meeting with all attendees. I recommend Zoom for remote conferences.
  4. Notify at least 10-15 minutes before the meeting if you’re unable to attend or will be late. Don’t postpone the meeting for a single person, because that ruins the schedule for all the other attendees.
  5. Appoint someone to record the meeting and transcribe it and store it somewhere where everyone can read it at a later time. This is good to remember what was decided and also for others to catch up on what was said and decided on the meeting.
  6. If you’re gonna treat attendees with some fruit / cookies / cake / etc, make sure that the remote attendees also gets treated something. (Call the nearest bakery or tell them to go buy something at the expense of the company.)
  7. Follow up on what was decided on the meeting after a previously decided time to implement.

Planning

Planning, planning and more planning! Everyone loves (or hates) planning. It’s even harder to do properly if you don’t use tools that suite your model of working. But it needs to be done.

So, tips on making planning easier for everyone, even your remote workers!

  1. Maintain a shared 1 year plan, 6 month plan, 3 month plan and a sprint plan. The plans should contain features, not tasks.
  2. Keep a shared backlog with all tasks, even tasks that are not connected to a feature. (Maintenance tasks, bug fixes, etc)
  3. Find the tools that allows you to follow the above structure.
  4. As an employee, maintain your own weekly Kanban board. My Kanban board consists of three columns: TODO, DOING, DONE. This works well for me. Whenever a task ends up in DONE, update it in the shared tool as well.
  5. Find a method, try it out at least 6 months, then adjust if needed. Switching methods of planning often is not fun.

Don’t apply my tips as law. Try them, adjust them and scrap them if needed. It’s what works for me.

Communication

Written communication is better than oral communication.

With that said, there are different kinds of written and oral documentation.

Chat

Chat is great! Chat sucks!

Chatting is a great tool if the message isn’t super important, it’s a nice place to vent of some steam. But, if important discussions are happening in the chatting tool, please summarize what was decided and put it in a blog post or wiki article.

Email

I love emails. I love email even more when the sender can write a proper email.

So what do I define as a proper email? Well, it needs a subject, preferably a well written one so I know at glance if I want to read this email now or if it can be read at a later time. Then it needs content. If you think the whole thing fits into the subject, don’t send an email, send a message in the chat application.

So, what is email good for? I find it useful for meeting summaries, announcements and different kinds of reports.

Requests and tasks should not be sent via email, it should be put into the planning tool and assigned to me.

SMS

SMS is great! It’s fast, works most of the time and usually gets the attention of the recipient. But! Use SMS with extreme caution. It should only be used in life or death situations (Servers are on fire, major parts of the product is down, etc). SMS should be reserved for alert systems like PagerDuty or similar.

Conference call (Skype, Zoom, etc)

Use this for meetings when there are remote attendees.

Phone call

As with SMS, use with extreme caution. Is quite nice to use for person to person communication if the call has been scheduled.

Blog

I love blog posts. They’re straightforward, usually structured and I can read them whenever it suits me. I wish more people used blogs when communicating on how to do stuff, announcements, reports and such things.

Wiki

A wiki is great for persistent documentation, stuff that you want to search and find easily when there’s a problem. It shouldn’t be filled with articles on how to setup Docker or how to brew coffee. That sort of stuff is better suited for a blog. What I do expect to find in a wiki are panic lists, documentation on how stuff is set up, where to find what info and more. That’s what I expect to find in a wiki. I want to visit the wiki when something is burning and be able to simply click my way around to find solutions.

Work hours

There’s so much to be said about work hours, but it pretty much boils down to this: Set a schedule, put it in a shared calendar and communicate whenever the reality differs from the schedule.

Off-work activities

Activities with your collegues can be great fun, but it requires planning, especially when there are remote workers. But in general, try to plan things to be done whenever the remote employees are present. It’s not fun to be the only one who always misses out on friday beers, go-karting and what not.

I think that’s all I want to write at the moment. Feel free to contact me on Twitter if you have any comments.

Ansible baby steps

Categories: ansible

A light introduction to Ansible, promoting best practices and install cowsay on Ubuntu 14.04.

What the hell is Ansible you ask? Well, I’m here to help!

Ansible is a tool to configure Linux (And Windows) servers via SSH and Python scripts. It allows you to write scripts in YAML and Python, which are executed against and on remote servers.

Why should you use Ansible? You should use Ansible if you want to avoid tedious and error prone manual work. Sure, it’s fine to run a few commands on your server to install a few applications, change some configuration files and so on, but fast forward a year, do you still remember what you did? Can you quickly run those commands again if you need a second server or need to replace the existing server?

If you answered yes on both both questions, Ansible isn’t for you. However, if you didn’t answer yes on both questions, tag along on my journey to teach you what Ansible is, how to use it for a single server and in large deployments.

Install Ansible

First things first, we need to install Ansible onto your computer, this is the control computer, the one that executes the commands on the remote targets. The target computers doesn’t need to have Ansible installed (But they do need Python installed!).

The easiest and recommended way of installing Ansible is via Pythons pip tool.

Run the following command to install Ansible via pip:

pip install --user ansible

This installs Ansible into my local Python library, which on my Mac OS X computer is located at /Users/myname/Library/Python/2.7/bin/ansible. Be sure to add /Users/myname/Library/Python/2.7/bin to your $PATH variable to be able to run Ansible properly.

Run ansible --version to verify you have a working installation.

Your first command

Let’s start off by pinging your computer:

ansible -m ping localhost

This will print something similar to this:

localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If you get a warning about a missing host file (/etc/ansible/hosts), just ignore it, we won’t use that file anyway.

The command we ran is Ansibles equivalent of running ping localhost, but it verifies that it can properly connect to the host. In the case of localhost, it should always work.

Ad-hoc commands

Remember I said earlier that Ansible is a bunch of scripts you can run against your targets? Well, Ansible can run stand-alone commands against your targets as well.

To print the current time according to your computer, run this:

ansible -a "date" localhost

The expected output looks something like this:

localhost | SUCCESS | rc=0 >>
Tue Jun 28 22:15:10 CEST 2016

You now know how to run ad-hoc commands against your computer.

Recap

Let’s recap what we’ve learnt so far.

  1. How to install Ansible
  2. How to make sure the connection to a target works
  3. How to display the current date and time according to a target

But let’s dig a bit deeper and try to understand what the parameters to the ansible command means.

  • -m = The module to run. A module in Ansible is a Python script to be executed on the target. The default module if none is specified is the shell module. It executes it’s arguments as a standard shell command.
  • -a = The module arguments. A module can accept zero or more arguments to decide what to do. In the case of figuring out the targets date, we used -a with a value of date but didn’t specify a module to run. This forwards date to the shell module, which runs the command.
  • localhost = The host pattern to match against. We used the full name of the host, but you can specify a regex as well, like this: db[0-9], which will try to connect to all hosts matching the regex. This however, requires an inventory file. More on that later.

Inventories

So, let’s talk about inventories. An inventory is an ini-like file which contains all your targets. It can look like this:

127.0.0.1

Altough that will work, it’s not very helpful. Let’s add a name to the host.

my-computer ansible_host=127.0.0.1

This gives is a nicer output, but it requires the remote target to have a user named the same as your local user. Let’s add that:

my-computer ansible_host=127.0.0.1 ansible_user=myname

Save the file somewhere, call it whatever. I usually create a directory for my project and create a folder called inventories inside that folder, then save my inventory file inside that directory. So I end up with something like this:

.
└── inventories
    └── development

My inventory file is called development.

Now we have a complete inventory. To add more targets, simply add a new line with the information needed.

Playbooks

The collection of scripts to be applied to a target are called a playbook in Ansible. Let’s make one!

Open your editor and type the following:

---
- hosts: my-computer
  tasks:
    - name: install cowsay
      apt: >
        name=cowsay
        update_cache=yes
      become: yes

And that’s your first playbook! Save it as playbook.yml in your project directory.

Let’s explain the parts of it.

- hosts: my-computer defines which hosts to apply the tasks to. This can contain the name of a host or a regex to match hosts. It can also be a group or the special group all which matches all hosts in an inventory.

tasks: defines a list of tasks to be executed from top to botton on the target.

- name: install cowsay is your first task. The name isn’t required, but highly recommended to have. You can name it whatever you want.

apt: > let’s Ansible know that we want to execute the apt module.

name=cowsay is the first argument to the apt module. It’s the name of the package we’d like to install. Different modules have different arguments.

update_cache=yes lets the apt know we want to run apt-get update before installing the package.

become: yes lets Ansible know that we want to run this module with sudo. So become: yes is equivalent to sudo my-module.

Now that we understand the playbook, let’s run it!

ansible-playbook -i inventory/development playbook.yml

We’re using a new command, ansible-playbook, which is what’s used to execute a playbook against targets.

The -i inventory/development tells Ansible to use our inventory file to create a collection of targets to execute the playbook against.

When you run this command, you should end up with something like this:

PLAY [my-computer] *************************************************************

TASK [setup] *******************************************************************
ok: [my-computer]

TASK [install cowsay] **********************************************************
changed: [my-computer]

PLAY RECAP *********************************************************************
my-computer                : ok=2    changed=1    unreachable=0    failed=0

This let’s us know that the task install cowsay ran and it changed something on the target. If you run the playbook again, it’ll say ok for that task instead of changed.

Run ansible -i inventory/development -a "cowsay" all to verify that cowsay was properly installed.

It should print something like this:

my-computer | SUCCESS | rc=0 >>
 __
<  >
 --
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Congratulations! You’ve created your first Ansible playbook and inventory.

That’s it for today.

Read more

As always, if you have a comment, don’t hesitate to reach out to me on Twitter @hagbarddenstore or via email hagbarddenstore@gmail.com or via IRC Freenode where I go by the name Kim^J.

Using nginx to load balance microservices

Categories: infrastructure

How to load balance services with nginx, confd and etcd.

Imagine this, you have a bunch of services running on a machine, all is fine and dandy. You have the occasional downtime when you upgrade a service to a new version, but nothing you can’t handle.

Then, out of nowhere, you get loads of traffic, you need to scale horizontally, adding more servers to your cluster. Upgrading becomes harder, takes longer and there’s more downtime.

You figure it’s time to load balance your services, but how to do it in a way that scales and is easy to manage?

Well, that’s where nginx+confd+etcd comes into play.

The overall architecture is this, nginx handles the load balancing part, confd updates nginx configuration based on values in etcd, and services update etcd with their information.

This allows confd to reconfigure nginx whenever there’s a change in etcd, thus reconfiguring nginx in near realtime.

So, how do we set this up? Well, I’m assuming you have the services part figured out already, so I’m gonna skip that part. So, onto how to get started with etcd.

Etcd

Etcd is a key-value database, with a simple HTTP API, allowing the services to easily insert values.

Etcd is very easy to install, it’s a matter of downloading an executable and running it. You can also run it on Docker using the quay.io/coreos/etcd image.

But we’re gonna focus on the stand-alone binary, so, head to https://github.com/coreos/etcd/releases/ and grab the latest stable release for your OS.

The package should be installed on atleast 3 different server nodes, preferably 5, to provide failover. Installation is easy, just unzip and run the executable by running the following command:

ETCD_DISCOVERY=$(curl https://discovery.etcd.io/new?size=3)

etcd --name $(hostname -s) \
     --initial-advertise-peer-urls http://10.0.0.1:2380 \
     --listen-peer-urls http://10.0.0.1:2380 \
     --listen-client-urls http://10.0.0.1:2379,http://127.0.0.1:2379 \
     --advertise-client-urls http://10.0.0.1:2379 \
     --discovery $ETCD_DISCOVERY

NOTE: The ETCD_DISCOVERY must be the same on all machines.

So, let’s explain the parameters.

  • –name = Name of the instance running etcd, this must be a within the cluster unique identifier. It’s used by etcd to separate nodes apart. The hostname or machine id are good candidates.
  • –initial-advertise-url = URL to advertise to other etcd nodes to allow internal etcd communication. The hostname or any IP address which other etcd nodes can reach are good values for this parameter.
  • –listen-peer-urls = URL on which etcd listens for internal etcd communication. This should be the same value as --initial-advertise-url in most cases.
  • –listen-client-urls = URLs on which etcd listens for client communication. This is the address you use to communicate with etcd. Preferably you want to add both 127.0.0.1 and a public IP, to allow both localhost communication and external clients.
  • –advertise-client-urls = URLs which etcd advertises to the cluster. This could be the same as --listen-client-urls minus the localhost address.
  • –discovery = URL to a discovery service, used by nodes to discover the cluster when no previous contact has been made. This value should be the same on all nodes you wish to include in the same cluster. You can get a new URL by running curl https://discovery.etcd.io/new?size=3 where 3 is the minimum amount of nodes in the cluster. You need to have atleast 3 nodes in your cluster to make a highly available cluster. A size between 5 and 9 is recommended if you’re running a cluster with high uptime requirements.

Now that we’ve got parameters covered, let’s run the command on atleast 3 servers and you should have a functional etcd cluster up and running. You can verify by running curl http://localhost:2379/version on one of the machines.

Next step is to setup nginx!

Nginx

We’re not gonna do any custom configuration on nginx, so a simple # apt-get install nginx is sufficient if you’re running a Debian-based OS.

With nginx running (Verify by running curl http://localhost/), let’s move on to the next step, which is installing and configuration confd!

Confd

So, finally at the step which does all of the magic!

First things first, we need to install confd. Head over to Github and get the latest release (At the time of writing, latest is v0.12.0-alpha3), put it on your nginx machines and unzip. Move the confd binary into /usr/bin/confd.

Next step is to create a configuration file for confd, a template and optionally an init startup script.

/etc/confd/conf.d/nginx.toml

This is the confd nginx configuration file, it tells confd where to find the template file, where to place the result, which command to run on change and what keys to watch.

[template]
src = "nginx.conf.tmpl"
dest = "/etc/nginx/sites-enabled/services.conf"
owner = "nginx"
mode = "0644"
keys = [
    "/services",
]
reload_cmd = "/usr/sbin/service nginx reload"
  • src = Name of the template file to execute on each change.
  • dest = Name of the file where the output of the template should be placed.
  • owner = File owner of the dest file.
  • mode = File mode of the dest file.
  • keys = Etcd keys to watch for change. You can watch /, but to ignore keys you’re not interested in, you should specify which keys you’re interested in. You don’t need to specify the full keys (That would defeat the point of this post!), but the static part in the beginning of the key, in this case /service.
  • reload_cmd = Command to run after the template has run.

Put the above content in /etc/confd/conf.d/nginx.toml, then continue with the next file.

/etc/confd/templates/nginx.conf.tmpl

Ah, the template file!

I’m not gonna explain the content of this file, it’s a mix between Go’s text/template markup and nginx’s configuration file.

If you want to figure out the stuff between {{ and }}, head over to Go’s text/template and confd template documentation.

{{ $services := ls "/services" }}
{{ range $service := $services }}
{{ $servers := getvs ( printf "/services/%s/servers/*" $service ) }}
{{ if $servers }}
upstream {{ $service }} {
  {{ range $server := $servers }}
  {{ $data := json $server }}
  server {{ $data.host }}:{{ $data.port }};
  {{ end }}
}
{{ end }}
{{ end }}

server {
  server_name hostname;
  
  {{ range $service := $services }}
  {{ $servers := getvs ( printf "/services/%s/servers/*" $service ) }}
  {{ if $servers }}
  location /{{ $service }} {
    rewrite /{{ $service }}/(.*) /$1 break;

    proxy_pass http://{{ $service }};
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto $scheme;
  }
  {{ end }}
  {{ end }}
}

Copy and paste the above content into /etc/confd/templates/nginx.conf.tmpl.

/etc/init/confd.conf

This step is optional and requires Upstart (Present on Ubuntu). Feel free to adapt the script to other init systems. The main part is the last line, which starts the confd daemon. Replace the etcd-address with the address to one of the machines running etcd.

If you wish to this as a daemon, without Upstart, simply run /usr/bin/confd -backend etcd -watch -node http://etcd-address:2379/ &.

description "confd daemon"

start on runlevel [12345]
stop on runlevel [!12345]

env DAEMON=/usr/bin/confd
env PID=/var/run/confd.pid

respawn
respawn limit 10 5

exec $DAEMON -backend etcd -watch -node http://etcd-address:2379/

If you went with the Upstart route, run sudo service confd start.

Onto the next step, we’re almost done!

Services

All of this did nothing! Calm your horses, we haven’t added any services to etcd yet!

This is quite simple though, simply execute

curl http://any-etcd-node:2379/v2/keys/services/$service_name$/servers/$hostname$ \
     -d value='{"host":"$public_ip$","port":$port$}' -d ttl=60 -X PUT

Replace $service_name$ with the name of the service you wish to load balance, replace $hostname$ with hostname or machine id of the instance running the service, replace $public_ip$ with the IP address on which the nginx machine(s) can reach the service and lastly replace $port$ with the port on which the service is listening for incoming HTTP traffic.

Do note the -d ttl=60 parameter, this tells etcd that it should delete the value in 60 seconds, so you need to continuously execute the curl command to keep the value in etcd. By doing this, you allow etcd to clean up services that is no longer available. Tweak the number to suit your use cases. My recommendation is to have a ttl of 60 and updating every 45 seconds. This allows for some downtime if a service crashes, but it shouldn’t affect things that much, but as said, adjust to your use case.

When you stop your service, you need to delete the key (And stop the updating script/routine!), this is done by running curl http://any-etcd-node:2379/v2/keys/services/$service_name$/servers/$hostname$ -X DELETE with the same values as the create script above. This tells etcd and confd that the service is no longer avilable and should be removed ASAP. My recommendation is to run this before you stop the service, as to hinder nginx from sending requests to a service which no longer exists.

Last words

This is it! I hope you enjoyed the post and that you’ll find it useful to setup your own highly available web applications.

Confd is by no means restricted to just configuring nginx, it can configure pretty much anything, as long as the configuration is file based and there’s a reload/restart command available.

I like using nginx since I know it quite well and it’s fast, mature and highly reliable.

Got any questions? Ping me on Twitter or send me a message on IRC where I go by the name Kim^J.

Uploading Ubuntu to Openstack Glance

Categories: openstack

How to upload Ubuntu 14.04 to an Openstack provider.

Run the following commands to download the latest Ubuntu 14.04 image and upload it to the Openstack provider.

$ curl \
    -o trusty-server-cloudimg-amd64-disk1.img \
    http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

$ glance image-create \
    --architecture x86_64 \
    --name "ubuntu-14.04.4" \
    --min-disk 10 \
    --os-version 14.04.4 \
    --disk-format qcow2 \
    --os-distro ubuntu \
    --min-ram 1024 \
    --container-format bare \
    --file trusty-server-cloudimg-amd64-disk1.img \
    --progress

The instructions assume you have a properly configured Openstack environment running on the computer where you run the above commands.

Uploading CoreOS to Openstack Glance

Categories: openstack

How to upload CoreOS to an Openstack provider.

Run the following commands to download the latest CoreOS image and upload it to the Openstack provider.

$ curl \
    -o coreos_production_openstack_image.img.bz2 \
    http://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2

$ bunzip2 coreos_production_openstack_image.img.bz2

$ glance image-create \
    --architecture x86_64 \
    --name "coreos" \
    --min-disk 10 \
    --disk-format qcow2 \
    --min-ram 1024 \
    --container-format bare \
    --file coreos_production_openstack_image.img \
    --progress

The instructions assume you have a properly configured Openstack environment running on the computer where you run the above commands.

Continuously deploy hugo sites

Categories: hugo

TL;DR;

How to setup continuous deployment of a Hugo website hosted on Github to AWS S3 by using Travis CI as the build/deployment service.

I finally did it. I setup something that builds my website and pushes it to AWS S3.

To be able to follow along, you need a Github account, an AWS account and you need to register on Travis-CI.

You should also have the Travis CI CLI installed to be able to encrypt values in the .travis.yml file.

I’m assuming that you have prior knowledge with AWS, Hugo and Git.

Creating S3 bucket

So, first you need an S3 bucket where you can host your content. Go ahead and create one and give it a unique name. My preference is to use the same name for the bucket as you would use for the domain which will point to the bucket.

Example would be naming the bucket example.com if your website URL is http://example.com.

So, with that done, onto the next task, making the bucket available to the world.

Set bucket policy

So, in the S3 console, navigate to your bucket, click on Properties, expand Permissions and click on Edit bucket policy.

Paste the following and change example.com to your buckets name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow Public Access to All Objects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::example.com/*"
        }
    ]
}

Click Save and that’s it! Your bucket is now available to the world. Onto the next task, which is allowing Travis CI to push data to your bucket.

Create IAM user

Head over to the IAM Console. In the left menu, click on Users, then click on Create New Users. Enter a username of your choice, I recommend travis-ci. Then click Create and then Download Credentials. Remember where you save the downloaded file, since you’ll need the values in that file later on.

Next up, allowing the newly created user access to the previously created S3 bucket.

Create bucket policy

Head over to Policies, click Create Policy, click Select next to Create Your Own Policy. Give the policy a name, like example.com-access or something nicer. Then write a description if you like. Then, copy the JSON below and paste it into the Policy Document textbox and replace example.com with the name of your bucket. Click Create Policy and you’re done!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1456258376000",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::example.com/*"
            ]
        }
    ]
}

Next, attaching the policy to the previously created user.

Attach policy to IAM user

So, head over to Users again and click on your Travis CI user. Click on the Permissions tab and click on Attach Policy and find your previously created policy, then click Attach Policy.

That’s it for the AWS part! Next is creating a Travis CI build script.

Creating Travis CI build script

So, to allow Travis CI do the magic, we need a build script.

Open your favorite text editor and paste the text below into your editor.

language: go
go:
  - 1.6

sudo: false

install:
  - go get github.com/spf13/hugo

script:
  - hugo

deploy:
  - provider: s3
    access_key_id: <access_key_id>
    secret_access_key: <secret_access_key>
    bucket: <bucket>
    region: eu-west-1
    local_dir: public
    skip_cleanup: true

Before we save, we need to change the <access_key_id>, <secret_access_key> and <bucket>.

To safely store the access key id, secret access key and bucket name, Travis CI provides you with a CLI tool to encrypt values. So, open up a terminal and write this:

cd path/to/your/hugo/git/repository

travis encrypt "<access_key_id_>"

This will ask you to confirm the repository and then return a value looking like this:

secure: "aaeeee..."

Simply copy that value and paste int your .travis.yml file one line below the key you’re setting the value for.

Repeat this for the <secret_access_key> and the <bucket> values.

The final file should look something like this:

language: go
go:
  - 1.6

sudo: false

install:
  - go get github.com/spf13/hugo

script:
  - hugo

deploy:
  - provider: s3
    access_key_id:
      secure: "gibberish"
    secret_access_key: 
      secure: "gibberish"
    bucket:
      secure: "gibberish"
    region: eu-west-1
    local_dir: public
    skip_cleanup: true

Save the file as .travis.yml, commit your changes and push to Github.

That’s it! Now all you need to do is to connect your Github repository on Travis CI, but that’s pretty straightforward, just head over to the Travis CI website and you’ll be guided.

Got any questions? Did I miss something, spelling errors or other suggestions? Fork and send me a pull request, or contact me on Twitter

Ansible gotcha: Playbook sudo vs task sudo

Categories: ansible

TL;DR;

When running a playbook with sudo: yes, it also runs the facts module with sudo, so ansible_user_dir will have the value of the root user, rather than the expected home directory of the ansible_ssh_user.

So, today I where pushing some generated SSH keys to servers to be able to pull changes from git without having to add each servers respective SSH key to the git server. Since I’m using Ansible to automate configuration of servers, I setup the task like this:

- name: push git deploy key
  copy:
    dest="{{ ansible_user_dir }}/.ssh/{{ item.name }}"
    content="{{ item.content }}"
  with_items:
    - name: id_rsa
      content: "{{ git_deploy_key_private }}"
    - name: id_rsa.pub
      content: "{{ git_deploy_key_public }}"
  no_log: True
  tags: configuration

Nothing wrong with the above code, I verified with ansible -m setup host that ansible_user_dir had the correct value. But, when you add sudo: yes to the playbook, like this:

- hosts: host
  sudo: yes
  roles:
    - { role: deploy_git_keys }

Then Ansible runs the facts module with sudo, which makes the ansible_user_dir contain the home directory of the user sudo was ran as, which by default is root. So, it had the value of /root rather than my expected /home/ubuntu. So the keys where pushed to /root/.ssh/ rather than my expected /home/ubuntu/.ssh/.

So, the solution is to not have sudo: yes in your playbook, but have them on the various tasks that really need it or to not rely on ansible_user_dir, but I prefer the former.

You can verify this behaviour by running the following playbook:

---
- hosts: all
  sudo: yes
  tasks:
    - name: print ansible_user_dir
      debug:
        var=ansible_user_dir
    - name: print $HOME
      command: echo $HOME

- hosts: all
  tasks:
    - name: print ansible_user_dir
      debug:
        var=ansible_user_dir
      sudo: yes
    - name: print $HOME
      command: echo $HOME
      sudo: yes

- hosts: all
  sudo: no
  tasks:
    - name: print ansible_user_dir
      debug:
        var=ansible_user_dir
    - name: print $HOME
      command: echo $HOME

The first play is gonna display /root twice, the second play will display /home/ubuntu first and then /root and the third play will show /home/ubuntu twice.