Archives For open source

It’s been one year after Cosmic started, forked off of the Apache CloudStack project. Cosmic 5.3 brings easy-to-use connectivity between VPCs. CloudStack users will recognise the Private Gateway functionality, which has been completely refactored in Cosmic 5.3. This blog shows what’s new.

The goal: connect VMs together that live in different VPCs
Basically, what you want is to setup a bunch of VPCs, create networks inside them where the VMs will live. Once you do that, there is no easy way to connect VMs from one VPC to another. Sure, you could make a VPN connection, and that is preferred in some cases. When you work in the same availability zone, and for example want Management VMs to access Acceptance or Production, there’s an easy alternative to get connectivity.

Create a Private Network
In Cosmic 5.3 you can create a network of type Private. A default network offering for this is created automatically, the only real difference is setting the type to Private (other networks are usually of type Isolated).

create-private-network-filled

Create Private Network

If you want, you can specify a VLAN of LSwitch manually by selecting an offering that supports it. You’d only do that in special cases, as it requires manual setup that is otherwise done automatically for you.

create-private-network-vlan

Create Private Network with a specific VLAN

Connect a VPC to the Private Network
Next, we’ll want to connect a VPC to this Private Network. That is done using a Private Gateway.

create-private-gw

Creating a Private Gateway

private-gw-connected

Private Gateway is connected to the Private Network

Now, the VPC is connected to the Private Network. You will want to repeat this step for all VPCs you want to connect. In my case I added 3 VPCs with each one Private Gateway. Make sure they have an unique ip address on the Private Network.

overview-of-vpcs

Overview of VPCs created

VPC1 has CIDR of 10.1.0.0/16 and has a Private Gateway using 172.16.0.1 connected to the Private Network. VPC2 has 172.16.0.2 and VPC3 has 172.16.0.3 as its ip address on the Private Network.

Making the VPCs aware of each other
By setting static routes to the other VPCs, to their ip address on the Private Network, it becomes easy to route traffic.

vpc-route-table

Setting routes to the other VPCs

VPC1 now knows that 10.2.0.0/16 is reachable via 172.16.0.2 (VPC2) and 10.3.0.0/16 is reachable via 172.16.0.3 (VPC3). If you fill the route table for VPC2 and VPC3 in the same way (create routes to the CIDRs of the other VPCs) then all VMS in these three VPCs can reach each other!

vms-in-vpcs

Three VMS in their own VPC

traffic-over-private-network

Inter-VPC traffic over the Private Network

Best of all, you can create this as a Domain Admin user.

Improvements over Apache CloudStack:

  • A special network type Private was created, to make a clear distinction between types Guest, Public and Private (lots of hacks in the code removed)
  • Able to use Domain Admin user, instead of ROOT user (even the RBAC in CloudStack cannot delegate the creation of Private Gateways to non-ROOT users).
  • Add a Private Gateway without pre-setting up a VLAN or Lswitch
  • Able to set Static Routes in the Route Table on the VPC level, rather than on the Private Gateway level. This allows setting routes to anything the kernel of the router VM accepts which is very flexible.

Conclusion
Cosmic 5.3 has made connecting VPCs together a whole lot easier. The fact that Domain Admins can now set this up (using Terraform & friends) makes it an easy to consume feature. Combined with the flexible static routes, one can basically implement any network design.

Get Cosmic for free now!

git_logoWhen contributing to open source projects, it’s pretty common these days to fork the project on Github, add your contribution, and then send your work as a so-called “pull request” to the project for inclusion. It’s nice, clean and fast. I did this last week to contribute to Apache CloudStack. When I wanted to contribute again today, I had to figure out how to get my “forked” repo up-to-date before I could send a new contribution.

Remember, you can read/write to your fork but only-read from the upstream repository.

Adding upstream as a remote
When you clone your forked repo to a local development machine, you get it setup like this:

git remote -v
origin git@github.com:remibergsma/cloudstack.git (fetch)
origin git@github.com:remibergsma/cloudstack.git (push)

As this refers to the “static” forked version, no new commits come in. For that to happen, we need to add the original repo as an extra “remote” that we’ll call “upstream”:

git remote add upstream https://github.com/apache/cloudstack

Now, run the same command again and you’ll see two:

git remote -v
origin git@github.com:remibergsma/cloudstack.git (fetch)
origin git@github.com:remibergsma/cloudstack.git (push)
upstream https://github.com/apache/cloudstack (fetch)
upstream https://github.com/apache/cloudstack (push)

The cloned git repo is now configured to both the forked and the upstream repo.

Let’s fetch the updates from upstream:

git fetch upstream

Sample output:

remote: Counting objects: 151, done.
remote: Compressing objects: 100% (123/123), done.
remote: Total 151 (delta 39), reused 0 (delta 0)
Receiving objects: 100% (151/151), 153.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (39/39), done.
From https://github.com/apache/cloudstack
   2f2ff4b..49cf2ac  4.4        -> upstream/4.4
   aca0f79..66b7738  4.5        -> upstream/4.5
* [new branch]      hotfix/4.4/CLOUDSTACK-8073 -> upstream/hotfix/4.4/CLOUDSTACK-8073
   85bb685..356793d  master     -> upstream/master
   b963bb1..36c0c38  volume-upload -> upstream/volume-upload

We now got the new updates in. Before you continue, be sure to be on the master branch:

git checkout master

Then we will rebase the new changes to our own master branch:

git rebase upstream/master

You can achieve the same by merging, but rebasing is usually cleaner and doesn’t add the extra merge commit.

Sample output:

Updating 4e1527e..356793d
Fast-forward
SSHKeyPairResponse.java                       | 12 ++++++++++++
SolidFireSharedPrimaryDataStoreLifeCycle.java | 33 +++++++++++++++++++++++++++++++++
RulesManagerImpl.java                         |  2 +-
ManagementServerImpl.java                     |  5 +----
4 files changed, 47 insertions(+), 5 deletions(-)

Finally, update your fork at Github with the new commits:

git push origin master

Branches other than master
Imagine you want to track another branch and sync that as well.

git checkout -b 4.5 origin/4.5

This will setup a local branch called ‘4.5’ that is linked to ‘origin/4.5’.

If you want to get them in sync again later on, the workflow is similar to above:

git checkout 4.5
git fetch upstream
git rebase upstream/4.5
git push origin 4.5

Automating this process
I wrote this script to synchronise my clones with upstream:


#!/bin/bash

# Sync upstream repo with fork repo
# Requires upstream repo to be defined

# Check if local repo is specified and exists
if [ -z $1 ]; then
 echo "Please specify repo to sync: $0 <dir>"
 exit 1
fi
if [ ! -d $1 ]; then
 echo "Dir $1 does not exist!"
 exit 1
fi

# Go into git repo and update
cd $1

# Check upstream
git remote -v | grep upstream >/dev/null 2>&1
RES=$?

if [ $RES -gt 0 ]; then
 echo "Upstream repo not defined. Please add it:
 git remote add http://github.com/..."
 exit 1
fi

# Update and push
git fetch upstream
git rebase upstream/master
git push origin master

Execute like this:

./update_origin_with_upstream.sh /path/to/git/repo

 

Happy contributing!