Cloud Computing at IFCA

1. Introduction

This is a beta service, since the deployment and development is ongoing. However, access is granted to certain users to test the functionality.

Check this document for updates frequenlty, since changes in the service may occur.

IFCA uses OpenStack for managing the cloud service, which provides several ways to access:

• Web dashboard

• OpenStack API, that can be used with the nova command line interface.

• Amazon EC2 API, available with the euca commands.

This documents focuses on the usage of the nova command in the GRIDUI Cluster.

2. Credentials

You need a valid account for accessing the cloud services. The username and password of that account may not be the same you are using in the GRIDUI Cluster. Contact the helpdesk if you need further information on your account.

Once you have a valid user and password, you can obtain your credentials in the dashboard of OpenStack. Login into Web portal and go to your settings area (Link in the upper right corner). There you two different kind of credentials:

• The OpenStack Credentials that can be used with the nova command

• The EC2 Credentials can be used with euca tools.

Download the configuration file for OpenStack and copy it to the GRIDUI before attempting any of the commands. Once you have the file, source it in order to have the environment ready.

$ . openrc.sh
Please enter your OpenStack Password:

Now you should be able to execute the commands to access the resources. Try that your environment is correct with the nova endpoints command. It should return all the services available in the OpenStack installation:

$ nova endpoints
+-------------+---------------------------------------------------------------+
| nova-volume | Value                                                         |
+-------------+---------------------------------------------------------------+
| adminURL    | http://cloud.ifca.es:8776/v1/c725b18d8d0643e7b410dc5a8d9ab554 |
| internalURL | http://cloud.ifca.es:8776/v1/c725b18d8d0643e7b410dc5a8d9ab554 |
| publicURL   | http://cloud.ifca.es:8776/v1/c725b18d8d0643e7b410dc5a8d9ab554 |
| region      | RegionOne                                                     |
+-------------+---------------------------------------------------------------+
+-------------+-------------------------------+
| glance      | Value                         |
+-------------+-------------------------------+
| adminURL    | http://glance.ifca.es:9292/v1 |
| internalURL | http://glance.ifca.es:9292/v1 |
| publicURL   | http://glance.ifca.es:9292/v1 |
| region      | RegionOne                     |
+-------------+-------------------------------+
+-------------+-----------------------------------------------------------------+
| nova        | Value                                                           |
+-------------+-----------------------------------------------------------------+
| adminURL    | http://cloud.ifca.es:8774/v1.1/c725b18d8d0643e7b410dc5a8d9ab554 |
| internalURL | http://cloud.ifca.es:8774/v1.1/c725b18d8d0643e7b410dc5a8d9ab554 |
| publicURL   | http://cloud.ifca.es:8774/v1.1/c725b18d8d0643e7b410dc5a8d9ab554 |
| region      | RegionOne                                                       |
| serviceName | nova                                                            |
+-------------+-----------------------------------------------------------------+
+-------------+------------------------------------------+
| ec2         | Value                                    |
+-------------+------------------------------------------+
| adminURL    | http://cloud.ifca.es:8773/services/Admin |
| internalURL | http://cloud.ifca.es:8773/services/Cloud |
| publicURL   | http://cloud.ifca.es:8773/services/Cloud |
| region      | RegionOne                                |
+-------------+------------------------------------------+
+-------------+------------------------------------+
| keystone    | Value                              |
+-------------+------------------------------------+
| adminURL    | http://keystone.ifca.es:35357/v2.0 |
| internalURL | http://keystone.ifca.es:5000/v2.0  |
| publicURL   | http://keystone.ifca.es:5000/v2.0  |
| region      | RegionOne                          |
+-------------+------------------------------------+

3. Creating machines

The cloud service lets you instantiate virtual machines (VM) on demand. When you request the creation of a new VM, you can select the operating systems and the size (RAM, Disk, CPUs) that will be used to run the machine. In this section we will show how to discover which software and sizes are available and how to start a new virtual machine.

3.1. Keypairs

Before attempting to start a new virtual machine, you should have a keypair that will allow you to login into the machine once it is running. Normally you just need to create one keypair that can be reused for all your virtual machines (although you can create as much SSH credentials as you want).

The nova keypair-list command shows your current keypairs. Initially the command should not return anything.

In order to create a new key, use nova keypair-add with a name for the key you want to use redirecting the ouput to the file where you want to store that key. For example, for creating a key named cloudkey that will be stored in cloudkey.pem:

$ nova keypair-add cloudkey > cloudkey.pem

Your recently created keypair should now appear in the list of available keypairs:

$ nova keypair-list
+----------+-------------------------------------------------+
| Name     | Fingerprint                                     |
+----------+-------------------------------------------------+
| cloudkey | 37:fd:b6:73:59:78:fd:f2:7f:e7:9c:1b:9a:88:a5:cb |
+----------+-------------------------------------------------+

Make sure that you keep safe the file cloudkey.pem, since it will contain the private key needed to access your cloud machines. Set proper permissions to the key before using it with chmod 600 clodkey.pem (only user can read or write). If you need to delete one of your keypairs, use the nova keypair-delete command.

3.2. Images

The service lets you run VMs with different Operating Systems, you can list all the available ones with the nova image-list command. The ID of the image will be used as arguments for other commands.

$ nova image-list
+--------------------------------------+----------------------------------------------+--------+--------+
| ID                                   | Name                                         | Status | Server |
+--------------------------------------+----------------------------------------------+--------+--------+
| 6b3046eb-4649-44d6-96c2-9a93d3aab8dc | Fedora 15                                    | ACTIVE |        |
| e803caa2-c247-4088-80fd-54e77b20a5cb | Fedora 15 initrd                             | ACTIVE |        |
| 6823e5b0-13fc-4ce3-afd8-057285820ed2 | Fedora 15 kernel                             | ACTIVE |        |
| 0249a9cc-dced-4c5f-91eb-d6900576206f | Fedora 17                                    | ACTIVE |        |
| f07c936f-7678-40e5-bbfd-f7142a5482ff | Fedora 17 initrd                             | ACTIVE |        |
| a0fbc138-1879-439f-8f78-9b98893778b3 | Fedora 17 kernel                             | ACTIVE |        |
| d3ac534d-d839-4b25-af92-c143930f3694 | Fedora 17 old glibc                          | ACTIVE |        |
| d1eec0f5-e948-435d-899c-d865320698d7 | IFCA Debian Wheezy (2011-08) JeOS            | ACTIVE |        |
| cdbb6f8f-d10e-4e2b-879d-250d29fb9dbb | IFCA Scientific Linux 5.5 + PROOF v5.30.00   | ACTIVE |        |
| 6857ee01-2ba9-4846-b788-9e826dd9aaba | IFCA Scientific Linux 5.5 + ROOT v5.30.00    | ACTIVE |        |
| 18d99a06-c3e5-4157-a0e3-37ec34bdfc24 | IFCA Scientific Linux 5.5 JeOS               | ACTIVE |        |
| 75896bad-05d3-45f6-9958-5940f82d0048 | IFCA Scientific Linux 6.2 JeOS               | ACTIVE |        |
| 486c139e-f34d-465c-959c-1b9c8bf60cfd | IFCA Ubuntu Server 10.04 JeOS                | ACTIVE |        |
| 694f2673-7ea3-4690-a25e-c9dd4297519a | IFCA Ubuntu Server 10.04 JeOS                | ACTIVE |        |
| 66963875-5389-4048-b385-6f7e12a0915f | IFCA Ubuntu Server 11.10 JeOS                | ACTIVE |        |
| 3ef6bb0c-6a17-47c9-a949-70256eb6651e | IFCA openSUSE 11.4 + Compilers               | ACTIVE |        |
| daaed27e-6226-4295-8018-ad3b6b5210f6 | IFCA openSUSE 11.4 + Compilers + Mathematica | ACTIVE |        |
| 29233856-ed8e-4b61-ac81-898eb5e7c263 | IFCA openSUSE 11.4 JeOS                      | ACTIVE |        |
| f4e39219-ad13-495e-a35b-315a94675b0f | Ubuntu 11.10 kernel                          | ACTIVE |        |
| 369455d3-7f84-4630-b60c-e0ebf29a410c | Ubuntu 12.04 JeOS                            | ACTIVE |        |
| 4590d3b0-1df6-49a7-ae68-4dde83089b01 | Ubuntu Server 11.10 JeOS                     | ACTIVE |        |
| fea1838f-29a0-47dd-bd84-6c6cc6806ff3 | cloudpipe                                    | ACTIVE |        |
| 6f02785c-5a39-4e1a-a7e3-75d48f0f0076 | ubuntu 12.04 kernel                          | ACTIVE |        |
+--------------------------------------+----------------------------------------------+--------+--------+

The nova image-show can give you more details about a given image, for example the "IFCA Scientific Linux 5.5 JeOS", which has an ID 18d99a06-c3e5-4157-a0e3-37ec34bdfc24:

$ nova image-show 18d99a06-c3e5-4157-a0e3-37ec34bdfc24
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| created  | 2012-01-30T10:12:22Z                 |
| id       | 18d99a06-c3e5-4157-a0e3-37ec34bdfc24 |
| minDisk  | 0                                    |
| minRam   | 0                                    |
| name     | IFCA Scientific Linux 5.5 JeOS       |
| progress | 100                                  |
| status   | ACTIVE                               |
| updated  | 2012-07-18T08:50:48Z                 |
+----------+--------------------------------------+

3.3. Sizes

As in the case of the image to use, you can select the size of the VM to start. The list of available sizes (flavors in OpenStack terminology) can be obtained with nova flavor-list:

$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| 1  | m1.tiny   | 512       | 0    | 0         |      | 1     | 1.0         | N/A       | {}          |
| 2  | m1.small  | 2048      | 10   | 20        |      | 1     | 1.0         | N/A       | {}          |
| 3  | m1.medium | 4096      | 10   | 40        |      | 2     | 1.0         | N/A       | {}          |
| 4  | m1.large  | 7000      | 10   | 80        |      | 4     | 1.0         | N/A       | {}          |
| 5  | m1.xlarge | 14000     | 10   | 160       |      | 8     | 1.0         | N/A       | {}          |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+

3.4. Starting a machine

For starting a new VM, you need to specify one image, one size and a name for the new machine. Optionally, you can also spcify a keypair (it is always recommended to do so). The nova boot command lets you start the machine. For example, for creating a VM that runs IFCA Scientific Linux 5.5 JeOS (ID 18d99a06-c3e5-4157-a0e3-37ec34bdfc24) of size m1.tiny with the key cloudkey and named testVM you would need to issue the following command:

$ nova boot --flavor m1.tiny --image 18d99a06-c3e5-4157-a0e3-37ec34bdfc24 --key-name cloudkey testVM
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig      | MANUAL                               |
| OS-EXT-STS:power_state | 0                                    |
| OS-EXT-STS:task_state  | scheduling                           |
| OS-EXT-STS:vm_state    | building                             |
| accessIPv4             |                                      |
| accessIPv6             |                                      |
| adminPass              | PGg4KxZo3Fn4                         |
| config_drive           |                                      |
| created                | 2012-09-28T10:02:02Z                 |
| flavor                 | m1.tiny                              |
| hostId                 |                                      |
| id                     | cd9e08b9-6899-4748-909a-2ff667ff1905 |
| image                  | IFCA Scientific Linux 5.5 JeOS       |
| key_name               | cloudkey                             |
| metadata               | {}                                   |
| name                   | testVM                               |
| progress               | 0                                    |
| status                 | BUILD                                |
| tenant_id              | c725b18d8d0643e7b410dc5a8d9ab554     |
| updated                | 2012-09-28T10:02:03Z                 |
| user_id                | db66762e4fe148f8b8484c461a7a7182     |
+------------------------+--------------------------------------+

The id of the machine will allow you to query its status with nova show:

$ nova show cd9e08b9-6899-4748-909a-2ff667ff1905
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| OS-DCF:diskConfig      | MANUAL                                                                |
| OS-EXT-STS:power_state | 2                                                                     |
| OS-EXT-STS:task_state  | None                                                                  |
| OS-EXT-STS:vm_state    | active                                                                |
| accessIPv4             |                                                                       |
| accessIPv6             |                                                                       |
| config_drive           |                                                                       |
| created                | 2012-09-28T10:02:02Z                                                  |
| flavor                 | m1.tiny (1)                                                           |
| hostId                 | 5ed92271869711d494f1326b9611825d5635ab659ea3e143c13ca8c6              |
| id                     | cd9e08b9-6899-4748-909a-2ff667ff1905                                  |
| image                  | IFCA Scientific Linux 5.5 JeOS (18d99a06-c3e5-4157-a0e3-37ec34bdfc24) |
| key_name               | cloudkey                                                              |
| metadata               | {}                                                                    |
| name                   | testVM                                                                |
| private network        | 172.16.2.8                                                            |
| progress               | 0                                                                     |
| status                 | ACTIVE                                                                |
| tenant_id              | c725b18d8d0643e7b410dc5a8d9ab554                                      |
| updated                | 2012-09-28T10:03:54Z                                                  |
| user_id                | db66762e4fe148f8b8484c461a7a7182                                      |
+------------------------+-----------------------------------------------------------------------+

Alternatively, you can use nova list to get the list of the current machines:

$ nova list
+--------------------------------------+--------------------------------------+--------+-------------------------------------+
| ID                                   | Name                                 | Status | Networks                            |
+--------------------------------------+--------------------------------------+--------+-------------------------------------+
| 4b999e84-b37e-4b95-952d-3414ba271930 | c725b18d8d0643e7b410dc5a8d9ab554-vpn | ACTIVE | private=172.16.2.2                  |
| 3051dea4-0164-4a3a-9af2-14efe7ea93e9 | horizon_test                         | ACTIVE | private=172.16.2.11, 193.146.75.142 |
| e02cee2d-c09f-4429-9724-91d7e10277ec | lbnl                                 | ACTIVE | private=172.16.2.10                 |
| cd9e08b9-6899-4748-909a-2ff667ff1905 | testVM                               | ACTIVE | private=172.16.2.8                  |
+--------------------------------------+--------------------------------------+--------+-------------------------------------+

3.5. Connecting to the machine

Once the machine status is ACTIVE, it will be ready for using it. You can connect via ssh with your key. The IP address of the machine is shown in the nova list output.

$ ssh -i cloudkey.pem root@172.16.2.8
Last login: Mon May 10 16:11:40 2010
[root@testvm ~]#

4. euca commands

This document is based in the nova native commands of OpenStack, but you can also use the euca- commands that use the EC2 API instead of the OpenStack one. The process for managing your VMs is practically identical. The euca commands use a different type of identifiers (ami)

The name of the machine can be

5. Create a machine

To create a machine you have to perform several steps:

• Decide which of the pre-built images you are going to use.
• Decide which with of the available sizes is suitable for you.
• Decide (and create if not ready) with keypair should be used to connect to the machine.

5.1. Image and size selection

5.1.1. Image selection

There are several pre-built images available. To check them, use the euca-describe-images command:

$ euca-describe-images
IMAGE   ami-00000008    None (cloudpipe)                available       public                  machine                 instance-store
IMAGE   ami-00000007    None (Debian Wheezy (2011-08))          available       public                  machine                 instance-store
IMAGE   ami-00000006    None (lucid-server-uec-amd64.img)               available       public                  machine                 instance-store
IMAGE   ami-00000003    None (Scientific Linux 5.5)             available       public                  machine                 instance-store
IMAGE   ami-00000001    None (Scientific Linux 5.5)             available       public                  machine                 instance-store

Once you have decided with image to use, write down its identifier (ami-XXXXXXXX).

5.1.2. Instance types

You can choose the size of your machine (i.e. how many CPUs and how much memory) from the following instance types:

5.1.2.1. Standard machines

5.1.2.2. High-memory machines

5.2. Create SSH credentials

For most of the users this is a one-time step (although you can create as much SSH credentials as you want). You have to create an SSH-keypair so as to inject it to the newly created machine with the following command (it will create a keypair named cloudkey and store it under ~/.cloud/cloudkey.pem):

$ euca-add-keypair cloudkey > ~/.cloud/cloudkey.pem

Make sure that you keep safe the file ~/.cloud/cloudkey.pem, since it will contain the private key needed to access your cloud machines. You can check the name later with the euca-describe-keypairs command.

5.3. Launching the instance

To launch the instance, you have to issue euca-run-instances, specifying:

• wich keypair to use (in the example cloudkey).

• wich size should be used (in the example m1.tiny).

• with image should be used (in the example ami-00000001).

$ euca-run-instances -k cloudkey -t m1.tiny ami-00000001
RESERVATION     r-1zdwog0m      ACES    default
INSTANCE        i-00000048      ami-00000001                    scheduling      cloudkey (ACES, None)   2011-09-02T12:19:41Z    None    None

You can check its status with euca-describe-instances

$ euca-describe-instances i-00000048
RESERVATION     r-vmfu1xq2      ACES    default
INSTANCE        i-00000048      ami-00000001    172.16.1.8      172.16.1.8      blocked         cloudkey (ACES, cloud01)        0       m1.tiny         2011-09-02T12:15:32Z    nova

5.4. Connect to the server

5.5. Authorize SSH connections and ping

If you decide not to use a VPN, but connect to your machines trough the GRIDUI cluster, you have to authorize such connections with:

$ euca-authorize -P tcp -p 22 default
$ euca-authorize -P icmp -t -1:-1 default

5.5.1. SSH Connection

You have to use the private identify file that you created before (~/.cloud/cloudkey.pem) and pass it to the SSH client. To check the IP to with you should connect, check it with euca-describe-instances

$ ssh -i ~/.cloud/cloudkey.pem root@172.16.1.8

5.6. Stopping the server

Images can be stopped with euca-terminate-instances

$ euca-terminate-instances i-00000048

6. Creating a Machine with OpenStack

Go to http://portal.cloud.ifca.es to access to OpenStack system, which lets you to create a new image in the cloud.

6.1. Image and size selection

You should launch the image that you want to use (within a list of OS) and click “Launch”. A new popup window will be shown and you have to choose the configuration of the system (requirements, name of the server...).

6.2. Create SSH credentials

You must import or create a new key in order to access to that image. To do so go to “Access & Security” tab and click on Create or Import Keypair.

6.3. Connect to the server

In order to access throw ssh to the image, you must asign an IP to the instance. Click on “Access & Security” again and select “Allocate IP to project”. Choose the type of IP that you want to use and click “Allocate IP”. After that, you need to link that IP with your new image. Click on the button “Associate IP” of your new IP and select the instance that you have just created.

6.3.1. SSH Connection

Last step is to download the keypair that you have created or imported and move it to the machine that you will use to conect to the isntance. Change permission to 600 and use the following command to connect:

$ Ssh -i clave.pem root@cloud.image.IP

Done

7. Using Cloud Storage

The storage in the cloud use volumes. Volumes are raw block devices that can be created dynamically with a desired size and associated with cloud images to be used as data disk. After use the data in the volume you can detach from the image and save for a later use of the persisted data.

7.1. Creating a Volume

To create a volume you have to run euca-create-volume command. For instance, to create a volume that is 100GB in size:

$ euca-create-volume -s 100 -z nova
VOLUME  vol-00000001    10      creating        2015-11-29

7.2. Using a Volume in an instance

You may attach block volumes to instances using euca-attach-volume. You will need to specify the local block device name (this will be used inside the instance) and the identified instance. Currently the devices to attach the volume should be /dev/xvdc, /dev/xvdd,... /dev/xvdz. Attaching volume vol-00000001 to image i-00000001 in device /dev/xvdc si done with:

$ euca-attach-volume -i i-00000001 -d /dev/xvdc vol-00000001

You can see the volume attached to the image with the command euca-describe-volumes.

$ euca-describe-volumes
VOLUME  vol-00000001     100            nova    in-use  2015-11-29
ATTACHMENT      vol-0000000c    i-00000051      /dev/xvdc

You can then use the new volume inside your running instance. As an example, the usage of the volume as an ext4 filesystem in a Ubuntu image is described below.

1. log into the image and check that the device is visible (as either root or as ubuntu user and use sudo for commands):

2. server-1 $ sudo fdisk -l | grep Disk
   Disk /dev/xvda doesn't contain a valid partition table
   Disk /dev/xvdb doesn't contain a valid partition table
   Disk /dev/xvdc doesn't contain a valid partition table
   Disk /dev/xvda: 10.7 GB, 10737418240 bytes
   Disk identifier: 0x00000000
   Disk /dev/xvdb: 21.5 GB, 21474836480 bytes
   Disk identifier: 0x00000000
   Disk /dev/xvdc: 107.4 GB, 107374182400 bytes
   Disk identifier: 0x00000000

3. Create a single ext4 partition on the device, an mount in the /srv mount point (-o sync is safe in case of image crashs):

4. server-1 $ sudo mkfs.ext4 /dev/xvdc
   (...)
   server-1 $ sudo mount -t ext4 -o sync /dev/xvdc /srv

5. Check that the volume is visible as a mounted filesystem:

6. server-1 $ df -h
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/xvda             9.9G  622M  8.8G   7% /
   none                  996M  144K  995M   1% /dev
   none                 1001M     0 1001M   0% /dev/shm
   none                 1001M   48K 1001M   1% /var/run
   none                 1001M     0 1001M   0% /var/lock
   none                 1001M     0 1001M   0% /lib/init/rw
   /dev/xvdb              20G  173M   19G   1% /mnt
   /dev/xvdc              99G  188M   94G   1% /srv

After you are done with the volume yo can detach from image with (you should umount it first on your instance):

$ euca-detach-volume vol-00000001

You must detach a volume before terminating an instance or deleting a volume. If you fail to detach a volume, it may leave the volume in an inconsistent state and you risk losing data.

7.3. Reusing an Old Volume

Attach with the new image:

$ euca-attach-volume -i i-00000002 -d /dev/xvdc vol-00000001

Because the filesystem is already created on the volume, you only need to mount it to access:

server-1 $ sudo mount -t ext4 -o sync /dev/xvdc /srv

After you are done with the volume yo can detach from image:

$ euca-detach-volume vol-00000001

7.4. Other uses of Volumes

With volume you can create snapshots of the data, recover it, delete volumes,...etc. More on volumes http://open.eucalyptus.com/wiki/Euca2oolsStorage

8. Advanced topics

8.1. Attach to the project's VPN

Each project has a VPN assigned to it. You can attach any computer to it, thus having it connected to your project's internal network. So as to do so, you have to perform several steps (instructions only for GNU/Linux):

1. Copy your ~.cloud to the machine that you want to attach to your project's VPN.

2. Install OpenVPN on that machine.

3. Launch openvpn with the nova-vpn.conf configuration file.

# cd cloud_credentials
# openvpn --config nova-vpn.conf

Please note that there are several paths in the nova-vpn.conf configuration file that are relative to the directory in which it is located. Should you wish to use different/separated paths, please edit nova-vpn.conf and adjust the cert, key and ca parameters.

MacOS users may use Tunnelblick (a GUI interface to OpenVPN) that can use the nova-vpn.conf and certificate files without any changes.

8.1.1. VPN with Ubuntu 10.04

1. Install network-manager-openvpn package

2. Add to /etc/dbus-1/system.d/nm-openvpn-service.conf between policy root and policy default:

<policy user=”at_console”>
<allow own=”org.freedesktop.NetworkManager.vpnc”/>
<allow send_destination=”org.freedesktop.NetworkManager.vpnc”/>
</policy>

3. With the network configuration in gnome bar, add new VPN conection importing nova-vpn.conf
4. Edit VPN conection, inside routing options, use this conection only for own resource.
5. Restart computer to get all changes in.

Now you can activate/deactivate VPN from gnome bar.

8.1.2. VPN with Windows

1. Install OpenVPN Connect Client from http://openvpn.net.

2. Rename nova-vpn.conf to nova-vpn.ovpn.

3. From Access -> Profiles -> Import from Local File, load the file nova-vpn.ovpn.

4. To connect, press on new nova-vpn button.

# === Manage multiple credentials === # and checking that NOVA_API, NOVA_CERT, NOVA_PROJECT, NOVA_URL, NOVA_USERNAME,