Published on 2018-01-30 10:41
Categories: aws
Tags: aws, china, cn-north-1, cn-northwest-1, amazon web services, amazon
TL;DR;
A collection of differences between non-China AWS and China AWS and how I’ve
solved them.
Recently I got my hands on a project which is to be deployed across multiple
regions in the world, one of them being China. Naive as I where, I thought
my CloudFormation stacks would just work in China, since they worked so
perfectly in both Europe (eu-west-1) and the US (us-east-1). But… I was
wrong. Very wrong.
After quite a lot of trial and error, I finally managed to get my modified
stack running in China, success!
So, to unburden the pain from you, I’ve compiled a list of all the caveats
I’ve found so far.
This blog post assumes you’re using some like CloudFormation, Terraform or
running commands via aws-cli. The experience will be different in the
AWS Console since it guides you to do the correct choices.
Instance types
This one is quite obvious, but still a bit of a pain.
China is lagging behind a bit when it comes to instance types, so please take
an extra look into your scripts before running them to ensure that the instance
types you’ve selected are available.
You can find the China instance types at https://www.amazonaws.cn/en/ec2/details/.
AMIs
This really shouldn’t be a surprise at all, since it’s different in all regions.
What caught my attention, was that some images (Ubuntu 16.04) aren’t up to date
to the current version, it even differs across the two regions in China.
Example: In cn-north-1 Ubuntu 16.04 20180109 is available, but in cn-northwest-1
it’s not. This is a bit weird, since 20180109 is available everywhere, but
cn-northwest-1.
I’ve written a small Python script to find all AMIs for Ubuntu 16.04 called
image-map.py and it contains this:
#!/usr/bin/env python2.7
import boto3
def main():
client = boto3.client("ec2")
for region in regions():
for image in images(region):
print "%s: %s" % (region, image.get("ImageId"))
def regions():
client = boto3.client("ec2")
for region in client.describe_regions().get("Regions", []):
yield region.get("RegionName", "")
def images(region_name):
client = boto3.client("ec2", region_name=region_name)
for image in client.describe_images(Filters = [ { "Name": "name", "Values": [ "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180109" ] } ]).get("Images", []):
yield image
if __name__ == "__main__":
main()
Replace ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180109 with
a string that matches your wanted AMI and then run the script twice, one with
a AWS “the world” account and one with a AWS China account. Example:
$ AWS_PROFILE=standard_aws_account ./image-map.py
$ AWS_PROFLE=aws_china_account ./image-map.py
This assumes you have setup two AWS profiles. Read more on
https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html.
IAM trust entities
Another thing that surprised me is that IAM trust entities sometimes are named
differently. The surprise stems from that I assumed the name of the entities to
be something that’s internal to AWS, in which case they are free to choose
whatever name they want, but this wasn’t the case.
The entities I use are monitoring.rds.amazonaws.com, ecs.amazonaws.com
and ec2.amazonaws.com. Both rds and ecs are named the same in China as they
are elsewhere, but ec2 is not. It’s called ec2.amazonaws.com.cn in China.
What I did in my CloudFormation template was to create a condition and use
Fn::If to select which entity name to use. Example:
Conditions:
China: !Or
- !Equals
- cn-north-1
- !Ref 'AWS::Region'
- !Equals
- cn-northwest-1
- !Ref 'AWS::Region'
Resources:
IamRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- !If
- China
- ec2.amazonaws.com.cn
- ec2.amazonaws.com
ManagedPolicyArns:
- !If
- China
- arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
Path: /
Type: AWS::IAM::Role
This snippet uses the same condition to select the correct policy ARN, since
they too are different in China.
There can be other entities that follow the same scheme, but I only came
across these since I mainly use RDS, ECS and EC2, but be aware of Lambda
and other services that heavily rely on IAM roles/profiles.
ARNs
As I just mentioned, ARNs are different in China as well, but they are quite
easy to manage, simple replace arn:aws: with arn:aws-cn: and you’re done!
Given we use the condition China which I defined in the snippet above, we
can select the correct ARN doing this:
- !If
- China
- arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
Cross-region VPC peering connections
No big surprise here, given that cross-region capability is quite new and
not introduced to that many regions yet, but it still made me a bit sad.
Non-mirrored package repositories
This caused the most pain of all issues. Package repositories aren’t mirrored
in China. The Ubuntu repositories are mirrored, so if you only install packages
from those repositories, you’re fine. But if you like me, run Docker or Gitlab,
you’re gonna have intermittent difficulties reaching the official mirrors.
There is a solution though, since companies inside China offers mirrors that
works.
So, for Docker, replace the official
deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
with a Chinese mirror provided by Alibaba
deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu xenial stable.
The GPG key can be fetched from http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg.
Same procedure is needed for Gitlab, replace the official
deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ xenial main with
deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu xenial main
provided by https://mirror.tuna.tsinghua.edu.cn/ (I guess this is some
kind of school). The official key can be used for this repository.
I haven’t found any need for other mirrors yet, but I’m sure there is a
need depending on which software you use.
No Route53
Again, no surprise, Route53 isn’t available in China. But, you can configure
Route53 with an AWS “world” account and use Route53 in China, since there are
edge nodes in China. You just can’t access the API in China via amazonaws.cn.
IAM isn’t global
IAM is supposed to be global, which it almost is, except China. China has their
own installation of IAM, so you need to duplicate your users from AWS into AWS
China. Not a big issue, but something to be aware of.
Final words
I think it’s amazing that we can use AWS in China, even with the limitations
and caveats. I think there’s still a lot for me to learn about China, so I’ll
probably revisit this post at a later time and update it with new problems and
solutions.
Published on 2018-01-23 11:39
Categories: ubuntu
Tags: ubuntu, gitlab
TL;DR;
A short description on how to install Gitlab Community Edition on Ubuntu 16.04
without using magic scripts.
If you, like me, dislike using magic scripts to install stuff, then this is the
post for you. If not, please read the official installation instructions on
https://about.gitlab.com/installation/#ubuntu.
Add Gitlabs APT key
First we need to add the GPG key. This key is used for both the official
package repository and the China mirror.
curl -L https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey | sudo apt-key add -
Add the Ubuntu 16.04 APT repository
To add the official repository to APT, run:
echo "deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ xenial main" | sudo tee /etc/apt/sources.list.d/gitlab.list
Bonus: APT mirror in China
If your server is located in China, there’s a great change it might not be
able to download Gitlab via the official mirrors. But, don’t fear, there are
mirrors inside China.
Run the following command, instead of the previous command to add the China
mirror:
echo "deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu xenial main" | sudo tee /etc/apt/sources.list.d/gitlab.list
No matter if you added the official repository or the China mirror, run the
following command to update APT:
sudo apt-get update -y
Install required packages
To install all packages needed by Gitlab, please run:
sudo apt-get install -y openssh-server ca-certificates gitlab-ce
The gitlab-ce package will install redis, nginx and postgresql as
needed by Gitlab. There is a package called gitlab which doesn’t include
these dependencies, but do tread carefully.
To setup Gitlab, run the following command:
sudo gitlab-ctl reconfigure
This will fire up Chef and run the included recipes for Gitlab, when done,
it will start all services needed for normal operation.
That’s it! Gitlab Community Edition and it’s dependencies are install. Visit
http://your_server_ip/ in your webrowser to setup the root password.
If you want to modify the installation (Perhaps to add HTTPS), please consult
the official documentation on https://docs.gitlab.com/omnibus/README.html
Published on 2017-01-10 13:20
Categories: ubuntu
Tags: ubuntu, debian, package management
TL;DR;
A quick tip on how to hold packages at a given version in Ubuntu. Will work
for other Debian based operating systems as well.
First, install the package at the given version. Here’s how to install
Elasticsearch 2.4.1:
sudo apt-get install -y elasticsearch=2.4.1
Verify it’s installed with the correct version by running:
dpkg -l | grep elasticsearch
This should return a line that looks like this:
hi elasticsearch 2.4.1 all Elasticsearch is a ...
Now it’s time to tell dpkg to hold this version:
echo "elasticsearch hold" | sudo dpkg --set-selections
Verify by running:
dpkg --get-selections | grep elasticsearch
That’s it! From now on, each sudo apt-get upgrade or equivalent won’t
upgrade the Elasticsearch package. Neat!
Published on 2016-10-18 15:26
Categories: life
Tags: remote work, life
I want to start by saying that I’m quite happy with my current employer, this
post isn’t written to speak ill of my employer, it’s just my view on working as
a remote employee at a company where the culture isn’t exactly remote friendly.
This post is going to be about the problems you can encounter as a remote
employee working for a company where most or all other employees are working at
the same office. I’ll try to suggest solutions to these problems, but not all
are tested, so implement with causion.
Meetings
While chatting and emailing are great tools, sometimes it’s better to be able to
talk to eachother and see eachother. That’s when you need to hold a meeting.
The best way to do this is to meet face to face, but this is quite impractical
if people are scattered across the globe.
Here’s my list of things to do before, during and after having a meeting:
- Write an agenda. Decide how much time to allocate for the meeting. Meetings
without an agenda and an end time are wasted meetings.
- Book a time when everyone needed for the meeting are able to attend. Put it
into a shared with an alert the day before and an hour before.
- Decide the conference tool to use and preferably test it before starting
the meeting. This can be tested between different attendees or a few minutes
before the meeting with all attendees. I recommend Zoom
for remote conferences.
- Notify at least 10-15 minutes before the meeting if you’re unable to attend
or will be late. Don’t postpone the meeting for a single person, because that
ruins the schedule for all the other attendees.
- Appoint someone to record the meeting and transcribe it and store it
somewhere where everyone can read it at a later time. This is good to remember
what was decided and also for others to catch up on what was said and decided
on the meeting.
- If you’re gonna treat attendees with some fruit / cookies / cake / etc, make
sure that the remote attendees also gets treated something. (Call the nearest
bakery or tell them to go buy something at the expense of the company.)
- Follow up on what was decided on the meeting after a previously decided time
to implement.
Planning
Planning, planning and more planning! Everyone loves (or hates) planning. It’s
even harder to do properly if you don’t use tools that suite your model of
working. But it needs to be done.
So, tips on making planning easier for everyone, even your remote workers!
- Maintain a shared 1 year plan, 6 month plan, 3 month plan and a sprint plan.
The plans should contain features, not tasks.
- Keep a shared backlog with all tasks, even tasks that are not connected to a
feature. (Maintenance tasks, bug fixes, etc)
- Find the tools that allows you to follow the above structure.
- As an employee, maintain your own weekly Kanban board. My Kanban board
consists of three columns: TODO, DOING, DONE. This works well for me. Whenever
a task ends up in DONE, update it in the shared tool as well.
- Find a method, try it out at least 6 months, then adjust if needed. Switching
methods of planning often is not fun.
Don’t apply my tips as law. Try them, adjust them and scrap them if needed. It’s
what works for me.
Communication
Written communication is better than oral communication.
With that said, there are different kinds of written and oral documentation.
Chat
Chat is great! Chat sucks!
Chatting is a great tool if the message isn’t super important, it’s a nice place
to vent of some steam. But, if important discussions are happening in the
chatting tool, please summarize what was decided and put it in a blog post or
wiki article.
Email
I love emails. I love email even more when the sender can write a proper email.
So what do I define as a proper email? Well, it needs a subject, preferably
a well written one so I know at glance if I want to read this email now or if
it can be read at a later time. Then it needs content. If you think the whole
thing fits into the subject, don’t send an email, send a message in the chat
application.
So, what is email good for? I find it useful for meeting summaries,
announcements and different kinds of reports.
Requests and tasks should not be sent via email, it should be put into the
planning tool and assigned to me.
SMS
SMS is great! It’s fast, works most of the time and usually gets the attention
of the recipient. But! Use SMS with extreme caution. It should only be used in
life or death situations (Servers are on fire, major parts of the product is
down, etc). SMS should be reserved for alert systems like
PagerDuty or similar.
Conference call (Skype, Zoom, etc)
Use this for meetings when there are remote attendees.
Phone call
As with SMS, use with extreme caution. Is quite nice to use for person to person
communication if the call has been scheduled.
Blog
I love blog posts. They’re straightforward, usually structured and I can read
them whenever it suits me. I wish more people used blogs when communicating on
how to do stuff, announcements, reports and such things.
Wiki
A wiki is great for persistent documentation, stuff that you want to search and
find easily when there’s a problem. It shouldn’t be filled with articles on how
to setup Docker or how to brew coffee. That sort of stuff is better suited for
a blog. What I do expect to find in a wiki are panic lists, documentation on
how stuff is set up, where to find what info and more. That’s what I expect to
find in a wiki. I want to visit the wiki when something is burning and be able
to simply click my way around to find solutions.
Work hours
There’s so much to be said about work hours, but it pretty much boils down to
this: Set a schedule, put it in a shared calendar and communicate whenever
the reality differs from the schedule.
Off-work activities
Activities with your collegues can be great fun, but it requires planning,
especially when there are remote workers. But in general, try to plan things
to be done whenever the remote employees are present. It’s not fun to be the
only one who always misses out on friday beers, go-karting and what not.
I think that’s all I want to write at the moment. Feel free to contact me on
Twitter if you have any comments.
Published on 2016-06-28 21:55
Categories: ansible
Tags: ansible
A light introduction to Ansible, promoting best practices and install cowsay
on Ubuntu 14.04.
What the hell is Ansible you ask? Well, I’m here to help!
Ansible is a tool to configure Linux (And Windows) servers via SSH and Python
scripts. It allows you to write scripts in
YAML and Python, which are executed
against and on remote servers.
Why should you use Ansible? You should use Ansible if you want to avoid
tedious and error prone manual work. Sure, it’s fine to run a few commands
on your server to install a few applications, change some configuration
files and so on, but fast forward a year, do you still remember what you did?
Can you quickly run those commands again if you need a second server or need
to replace the existing server?
If you answered yes on both both questions, Ansible isn’t for you. However, if
you didn’t answer yes on both questions, tag along on my journey to teach you
what Ansible is, how to use it for a single server and in large deployments.
Install Ansible
First things first, we need to install Ansible onto your computer, this is the
control computer, the one that executes the commands on the remote targets. The
target computers doesn’t need to have Ansible installed (But they do need
Python installed!).
The easiest and recommended way of installing Ansible is via Pythons
pip tool.
Run the following command to install Ansible via pip:
pip install --user ansible
This installs Ansible into my local Python library, which on my Mac OS X
computer is located at /Users/myname/Library/Python/2.7/bin/ansible. Be
sure to add /Users/myname/Library/Python/2.7/bin to your $PATH variable
to be able to run Ansible properly.
Run ansible --version to verify you have a working installation.
Your first command
Let’s start off by pinging your computer:
ansible -m ping localhost
This will print something similar to this:
localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}
If you get a warning about a missing host file (/etc/ansible/hosts), just
ignore it, we won’t use that file anyway.
The command we ran is Ansibles equivalent of running ping localhost, but
it verifies that it can properly connect to the host. In the case of
localhost, it should always work.
Ad-hoc commands
Remember I said earlier that Ansible is a bunch of scripts you can run
against your targets? Well, Ansible can run stand-alone commands against
your targets as well.
To print the current time according to your computer, run this:
ansible -a "date" localhost
The expected output looks something like this:
localhost | SUCCESS | rc=0 >>
Tue Jun 28 22:15:10 CEST 2016
You now know how to run ad-hoc commands against your computer.
Recap
Let’s recap what we’ve learnt so far.
- How to install Ansible
- How to make sure the connection to a target works
- How to display the current date and time according to a target
But let’s dig a bit deeper and try to understand what the parameters to
the ansible command means.
-m = The module to run. A module in Ansible is a Python script to be
executed on the target. The default module if none is specified is
the shell module. It executes it’s arguments as a standard shell command.-a = The module arguments. A module can accept zero or more arguments to
decide what to do. In the case of figuring out the targets date, we used
-a with a value of date but didn’t specify a module to run. This
forwards date to the shell module, which runs the command.- localhost = The host pattern to match against. We used the full name of the
host, but you can specify a regex as well, like this:
db[0-9], which
will try to connect to all hosts matching the regex. This however, requires
an inventory file. More on that later.
Inventories
So, let’s talk about inventories. An inventory is an ini-like file which
contains all your targets. It can look like this:
127.0.0.1
Altough that will work, it’s not very helpful. Let’s add a name to the host.
my-computer ansible_host=127.0.0.1
This gives is a nicer output, but it requires the remote target to have a
user named the same as your local user. Let’s add that:
my-computer ansible_host=127.0.0.1 ansible_user=myname
Save the file somewhere, call it whatever. I usually create a directory for
my project and create a folder called inventories inside that folder, then
save my inventory file inside that directory. So I end up with something
like this:
.
└── inventories
└── development
My inventory file is called development.
Now we have a complete inventory. To add more targets, simply add a new
line with the information needed.
Playbooks
The collection of scripts to be applied to a target are called a playbook in
Ansible. Let’s make one!
Open your editor and type the following:
---
- hosts: my-computer
tasks:
- name: install cowsay
apt: >
name=cowsay
update_cache=yes
become: yes
And that’s your first playbook! Save it as playbook.yml in your project directory.
Let’s explain the parts of it.
- hosts: my-computer defines which hosts to apply the tasks to. This can
contain the name of a host or a regex to match hosts. It can also be a group
or the special group all which matches all hosts in an inventory.
tasks: defines a list of tasks to be executed from top to botton on the
target.
- name: install cowsay is your first task. The name isn’t required, but
highly recommended to have. You can name it whatever you want.
apt: > let’s Ansible know that we want to execute the apt module.
name=cowsay is the first argument to the apt module. It’s the name of
the package we’d like to install. Different modules
have different arguments.
update_cache=yes lets the apt know we want to run apt-get update before
installing the package.
become: yes lets Ansible know that we want to run this module with sudo.
So become: yes is equivalent to sudo my-module.
Now that we understand the playbook, let’s run it!
ansible-playbook -i inventory/development playbook.yml
We’re using a new command, ansible-playbook, which is what’s used to
execute a playbook against targets.
The -i inventory/development tells Ansible to use our inventory file
to create a collection of targets to execute the playbook against.
When you run this command, you should end up with something like this:
PLAY [my-computer] *************************************************************
TASK [setup] *******************************************************************
ok: [my-computer]
TASK [install cowsay] **********************************************************
changed: [my-computer]
PLAY RECAP *********************************************************************
my-computer : ok=2 changed=1 unreachable=0 failed=0
This let’s us know that the task install cowsay ran and it changed something
on the target. If you run the playbook again, it’ll say ok for that task
instead of changed.
Run ansible -i inventory/development -a "cowsay" all to verify that cowsay
was properly installed.
It should print something like this:
my-computer | SUCCESS | rc=0 >>
__
< >
--
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Congratulations! You’ve created your first Ansible playbook and inventory.
That’s it for today.
Read more
As always, if you have a comment, don’t hesitate to reach out to me on
Twitter @hagbarddenstore or via
email hagbarddenstore@gmail.com or
via IRC Freenode where I go by the name Kim^J.
Published on 2016-03-11 00:35
Categories: infrastructure
Tags: etcd, skydns, nginx, load balance, services, service discovery
How to load balance services with nginx, confd and etcd.
Imagine this, you have a bunch of services running on a machine, all is fine
and dandy. You have the occasional downtime when you upgrade a service to a new
version, but nothing you can’t handle.
Then, out of nowhere, you get loads of traffic, you need to scale horizontally,
adding more servers to your cluster. Upgrading becomes harder, takes longer
and there’s more downtime.
You figure it’s time to load balance your services, but how to do it in a way
that scales and is easy to manage?
Well, that’s where nginx+confd+etcd comes into play.
The overall architecture is this, nginx handles the load balancing part, confd
updates nginx configuration based on values in etcd, and services update etcd
with their information.
This allows confd to reconfigure nginx whenever there’s a change in etcd, thus
reconfiguring nginx in near realtime.
So, how do we set this up? Well, I’m assuming you have the services part
figured out already, so I’m gonna skip that part. So, onto how to get started
with etcd.
Etcd
Etcd is a key-value database, with a simple HTTP API, allowing the services to
easily insert values.
Etcd is very easy to install, it’s a matter of downloading an executable and
running it. You can also run it on Docker using the quay.io/coreos/etcd
image.
But we’re gonna focus on the stand-alone binary, so, head to
https://github.com/coreos/etcd/releases/ and grab the latest stable release for
your OS.
The package should be installed on atleast 3 different server nodes, preferably
5, to provide failover. Installation is easy, just unzip and run the executable
by running the following command:
ETCD_DISCOVERY=$(curl https://discovery.etcd.io/new?size=3)
etcd --name $(hostname -s) \
--initial-advertise-peer-urls http://10.0.0.1:2380 \
--listen-peer-urls http://10.0.0.1:2380 \
--listen-client-urls http://10.0.0.1:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://10.0.0.1:2379 \
--discovery $ETCD_DISCOVERY
NOTE: The ETCD_DISCOVERY must be the same on all machines.
So, let’s explain the parameters.
- –name = Name of the instance running etcd, this must be a within the
cluster unique identifier. It’s used by etcd to separate nodes apart. The
hostname or machine id are good candidates.
- –initial-advertise-url = URL to advertise to other etcd nodes to allow
internal etcd communication. The hostname or any IP address which other etcd
nodes can reach are good values for this parameter.
- –listen-peer-urls = URL on which etcd listens for internal etcd
communication. This should be the same value as
--initial-advertise-url in
most cases.
- –listen-client-urls = URLs on which etcd listens for client
communication. This is the address you use to communicate with etcd.
Preferably you want to add both
127.0.0.1 and a public IP, to allow both
localhost communication and external clients.
- –advertise-client-urls = URLs which etcd advertises to the cluster. This
could be the same as
--listen-client-urls minus the localhost address.
- –discovery = URL to a discovery service, used by nodes to discover the
cluster when no previous contact has been made. This value should be the same
on all nodes you wish to include in the same cluster. You can get a new URL
by running
curl https://discovery.etcd.io/new?size=3 where 3 is the
minimum amount of nodes in the cluster. You need to have atleast 3 nodes in
your cluster to make a highly available cluster. A size between 5 and 9 is
recommended if you’re running a cluster with high uptime requirements.
Now that we’ve got parameters covered, let’s run the command on atleast 3
servers and you should have a functional etcd cluster up and running. You can
verify by running curl http://localhost:2379/version on one of the machines.
Next step is to setup nginx!
Nginx
We’re not gonna do any custom configuration on nginx, so a simple
# apt-get install nginx is sufficient if you’re running a Debian-based OS.
With nginx running (Verify by running curl http://localhost/), let’s move
on to the next step, which is installing and configuration confd!
Confd
So, finally at the step which does all of the magic!
First things first, we need to install confd. Head over to
Github and get the latest
release (At the time of writing, latest is v0.12.0-alpha3), put it on your
nginx machines and unzip. Move the confd binary into /usr/bin/confd.
Next step is to create a configuration file for confd, a template and
optionally an init startup script.
/etc/confd/conf.d/nginx.toml
This is the confd nginx configuration file, it tells confd where to find the
template file, where to place the result, which command to run on change and
what keys to watch.
[template]
src = "nginx.conf.tmpl"
dest = "/etc/nginx/sites-enabled/services.conf"
owner = "nginx"
mode = "0644"
keys = [
"/services",
]
reload_cmd = "/usr/sbin/service nginx reload"
- src = Name of the template file to execute on each change.
- dest = Name of the file where the output of the template should be
placed.
- owner = File owner of the dest file.
- mode = File mode of the dest file.
- keys = Etcd keys to watch for change. You can watch
/, but to ignore
keys you’re not interested in, you should specify which keys you’re
interested in. You don’t need to specify the full keys (That would defeat
the point of this post!), but the static part in the beginning of the key, in
this case /service.
- reload_cmd = Command to run after the template has run.
Put the above content in /etc/confd/conf.d/nginx.toml, then continue with
the next file.
/etc/confd/templates/nginx.conf.tmpl
Ah, the template file!
I’m not gonna explain the content of this file, it’s a mix between Go’s
text/template markup and nginx’s configuration file.
If you want to figure out the stuff between {{ and }}, head over to
Go’s text/template and
confd template documentation.
{{ $services := ls "/services" }}
{{ range $service := $services }}
{{ $servers := getvs ( printf "/services/%s/servers/*" $service ) }}
{{ if $servers }}
upstream {{ $service }} {
{{ range $server := $servers }}
{{ $data := json $server }}
server {{ $data.host }}:{{ $data.port }};
{{ end }}
}
{{ end }}
{{ end }}
server {
server_name hostname;
{{ range $service := $services }}
{{ $servers := getvs ( printf "/services/%s/servers/*" $service ) }}
{{ if $servers }}
location /{{ $service }} {
rewrite /{{ $service }}/(.*) /$1 break;
proxy_pass http://{{ $service }};
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forward-Proto $scheme;
}
{{ end }}
{{ end }}
}
Copy and paste the above content into /etc/confd/templates/nginx.conf.tmpl.
/etc/init/confd.conf
This step is optional and requires Upstart (Present on Ubuntu). Feel free to
adapt the script to other init systems. The main part is the last line, which
starts the confd daemon. Replace the etcd-address with the address to one
of the machines running etcd.
If you wish to this as a daemon, without Upstart, simply run
/usr/bin/confd -backend etcd -watch -node http://etcd-address:2379/ &.
description "confd daemon"
start on runlevel [12345]
stop on runlevel [!12345]
env DAEMON=/usr/bin/confd
env PID=/var/run/confd.pid
respawn
respawn limit 10 5
exec $DAEMON -backend etcd -watch -node http://etcd-address:2379/
If you went with the Upstart route, run sudo service confd start.
Onto the next step, we’re almost done!
Services
All of this did nothing! Calm your horses, we haven’t added any services to
etcd yet!
This is quite simple though, simply execute
curl http://any-etcd-node:2379/v2/keys/services/$service_name$/servers/$hostname$ \
-d value='{"host":"$public_ip$","port":$port$}' -d ttl=60 -X PUT
Replace $service_name$ with the name of the service you wish to load balance,
replace $hostname$ with hostname or machine id of the instance running the
service, replace $public_ip$ with the IP address on which the nginx
machine(s) can reach the service and lastly replace $port$ with the port on
which the service is listening for incoming HTTP traffic.
Do note the -d ttl=60 parameter, this tells etcd that it should delete the
value in 60 seconds, so you need to continuously execute the curl command to
keep the value in etcd. By doing this, you allow etcd to clean up services that
is no longer available. Tweak the number to suit your use cases. My
recommendation is to have a ttl of 60 and updating every 45 seconds. This
allows for some downtime if a service crashes, but it shouldn’t affect things
that much, but as said, adjust to your use case.
When you stop your service, you need to delete the key (And stop the updating
script/routine!), this is done by running
curl http://any-etcd-node:2379/v2/keys/services/$service_name$/servers/$hostname$ -X DELETE
with the same values as the create script above. This tells etcd and confd that
the service is no longer avilable and should be removed ASAP. My recommendation
is to run this before you stop the service, as to hinder nginx from sending
requests to a service which no longer exists.
Last words
This is it! I hope you enjoyed the post and that you’ll find it useful to setup
your own highly available web applications.
Confd is by no means restricted to just configuring nginx, it can configure
pretty much anything, as long as the configuration is file based and there’s a
reload/restart command available.
I like using nginx since I know it quite well and it’s fast, mature and highly
reliable.
Got any questions? Ping me on Twitter or
send me a message on IRC where I go by the
name Kim^J.
Published on 2016-03-03 16:28
Categories: openstack
Tags: ubuntu, glance, openstack
How to upload Ubuntu 14.04 to an Openstack provider.
Run the following commands to download the latest Ubuntu 14.04 image and
upload it to the Openstack provider.
$ curl \
-o trusty-server-cloudimg-amd64-disk1.img \
http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
$ glance image-create \
--architecture x86_64 \
--name "ubuntu-14.04.4" \
--min-disk 10 \
--os-version 14.04.4 \
--disk-format qcow2 \
--os-distro ubuntu \
--min-ram 1024 \
--container-format bare \
--file trusty-server-cloudimg-amd64-disk1.img \
--progress
The instructions assume you have a properly configured Openstack environment
running on the computer where you run the above commands.
Published on 2016-03-03 16:27
Categories: openstack
Tags: coreos, glance, openstack
How to upload CoreOS to an Openstack provider.
Run the following commands to download the latest CoreOS image and upload it
to the Openstack provider.
$ curl \
-o coreos_production_openstack_image.img.bz2 \
http://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
$ bunzip2 coreos_production_openstack_image.img.bz2
$ glance image-create \
--architecture x86_64 \
--name "coreos" \
--min-disk 10 \
--disk-format qcow2 \
--min-ram 1024 \
--container-format bare \
--file coreos_production_openstack_image.img \
--progress
The instructions assume you have a properly configured Openstack environment
running on the computer where you run the above commands.
Published on 2016-02-24 22:14
Categories: hugo
Tags: hugo, s3, aws, travis-ci, continuous deployment
TL;DR;
How to setup continuous deployment of a Hugo website hosted on Github to AWS S3
by using Travis CI as the build/deployment service.
I finally did it. I setup something that builds my website and pushes it to AWS
S3.
To be able to follow along, you need a Github
account, an AWS account and you need to register on
Travis-CI.
You should also have the Travis CI CLI installed to be able to encrypt values
in the .travis.yml file.
I’m assuming that you have prior knowledge with AWS, Hugo and Git.
Creating S3 bucket
So, first you need an S3 bucket where you can host your content. Go ahead and
create one and give it a unique name. My preference is to use the same name
for the bucket as you would use for the domain which will point to the bucket.
Example would be naming the bucket example.com if your website URL is
http://example.com.
So, with that done, onto the next task, making the bucket available to the world.
Set bucket policy
So, in the S3 console, navigate to your bucket, click on Properties, expand
Permissions and click on Edit bucket policy.
Paste the following and change example.com to your buckets name:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Allow Public Access to All Objects",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::example.com/*"
}
]
}
Click Save and that’s it! Your bucket is now available to the world. Onto the
next task, which is allowing Travis CI to push data to your bucket.
Create IAM user
Head over to the IAM Console. In the left menu, click on Users, then click
on Create New Users. Enter a username of your choice, I recommend travis-ci.
Then click Create and then Download Credentials. Remember where you save
the downloaded file, since you’ll need the values in that file later on.
Next up, allowing the newly created user access to the previously created S3
bucket.
Create bucket policy
Head over to Policies, click Create Policy, click Select next to
Create Your Own Policy. Give the policy a name, like example.com-access or
something nicer. Then write a description if you like. Then, copy the JSON below
and paste it into the Policy Document textbox and replace example.com with
the name of your bucket. Click Create Policy and you’re done!
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1456258376000",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::example.com/*"
]
}
]
}
Next, attaching the policy to the previously created user.
Attach policy to IAM user
So, head over to Users again and click on your Travis CI user. Click on the
Permissions tab and click on Attach Policy and find your previously created
policy, then click Attach Policy.
That’s it for the AWS part! Next is creating a Travis CI build script.
Creating Travis CI build script
So, to allow Travis CI do the magic, we need a build script.
Open your favorite text editor and paste the text below into your editor.
language: go
go:
- 1.6
sudo: false
install:
- go get github.com/spf13/hugo
script:
- hugo
deploy:
- provider: s3
access_key_id: <access_key_id>
secret_access_key: <secret_access_key>
bucket: <bucket>
region: eu-west-1
local_dir: public
skip_cleanup: true
Before we save, we need to change the <access_key_id>, <secret_access_key>
and <bucket>.
To safely store the access key id, secret access key and bucket name, Travis CI
provides you with a CLI tool to encrypt values. So, open up a terminal and write
this:
cd path/to/your/hugo/git/repository
travis encrypt "<access_key_id_>"
This will ask you to confirm the repository and then return a value looking like
this:
secure: "aaeeee..."
Simply copy that value and paste int your .travis.yml file one line below the
key you’re setting the value for.
Repeat this for the <secret_access_key> and the <bucket> values.
The final file should look something like this:
language: go
go:
- 1.6
sudo: false
install:
- go get github.com/spf13/hugo
script:
- hugo
deploy:
- provider: s3
access_key_id:
secure: "gibberish"
secret_access_key:
secure: "gibberish"
bucket:
secure: "gibberish"
region: eu-west-1
local_dir: public
skip_cleanup: true
Save the file as .travis.yml, commit your changes and push to Github.
That’s it! Now all you need to do is to connect your Github repository on
Travis CI, but that’s pretty straightforward, just head over to the Travis CI
website and you’ll be guided.
Got any questions? Did I miss something, spelling errors or other suggestions?
Fork and
send me a pull request, or contact me on Twitter
Published on 2015-11-11 14:33
Categories: ansible
Tags: ansible, ops
TL;DR;
When running a playbook with sudo: yes, it also runs the facts module
with sudo, so ansible_user_dir will have the value of the root user, rather
than the expected home directory of the ansible_ssh_user.
So, today I where pushing some generated SSH keys to servers to be able to pull
changes from git without having to add each servers respective SSH key to the
git server. Since I’m using Ansible to automate
configuration of servers, I setup the task like this:
- name: push git deploy key
copy:
dest="{{ ansible_user_dir }}/.ssh/{{ item.name }}"
content="{{ item.content }}"
with_items:
- name: id_rsa
content: "{{ git_deploy_key_private }}"
- name: id_rsa.pub
content: "{{ git_deploy_key_public }}"
no_log: True
tags: configuration
Nothing wrong with the above code, I verified with ansible -m setup host that
ansible_user_dir had the correct value. But, when you add sudo: yes to the
playbook, like this:
- hosts: host
sudo: yes
roles:
- { role: deploy_git_keys }
Then Ansible runs the facts module with sudo, which
makes the ansible_user_dir contain the home directory of the user sudo was
ran as, which by default is root. So, it had the value of /root rather
than my expected /home/ubuntu. So the keys where pushed to /root/.ssh/
rather than my expected /home/ubuntu/.ssh/.
So, the solution is to not have sudo: yes in your playbook, but have them on
the various tasks that really need it or to not rely on ansible_user_dir, but
I prefer the former.
You can verify this behaviour by running the following playbook:
---
- hosts: all
sudo: yes
tasks:
- name: print ansible_user_dir
debug:
var=ansible_user_dir
- name: print $HOME
command: echo $HOME
- hosts: all
tasks:
- name: print ansible_user_dir
debug:
var=ansible_user_dir
sudo: yes
- name: print $HOME
command: echo $HOME
sudo: yes
- hosts: all
sudo: no
tasks:
- name: print ansible_user_dir
debug:
var=ansible_user_dir
- name: print $HOME
command: echo $HOME
The first play is gonna display /root twice, the second play will display
/home/ubuntu first and then /root and the third play will show
/home/ubuntu twice.