天天看點

在ubuntu上安裝openstack的swift元件-installing openstack object storage

這幾天正在自己搭建一個swift的環境。看了一些資料,結合自己的實踐,寫下了它:

1.實體環境說明:

Linux系統版本:Ubuntu Server 12.04
Proxy Server IP:   		10.214.0.181
Storage Server One:		10.214.0.179
Storage Server Two:		10.214.0.180
Storage Server three:		10.214.0.182
           

2.重要元件說明:

node:運作一個或多個object storage service

proxy node:運作 proxy services

auth node:運作Auth service

storage:運作account,container,and object services

Ring:是Openstack object storage 到實體裝置的映射集合

在ubuntu上安裝openstack的swift元件-installing openstack object storage

在所有節點上的準備工作:

安裝服務

apt-get install swift openssh-server rsync memcached python-netifaces python-xattr python-memcache
           

配置路徑

mkdir -p /etc/swift
chown -R swift:swift /etc/swift/
           

建立檔案 vim /etc/swift/swift.conf

[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_suffix = fLIbertYgibbitZ
           

3.安裝和配置storage nodes

安裝:Storage node packages,所有storage都要進行操作。

aptitude install swift-account swift-container swift-object xfsprogs
           

建立xfs系統分區,當然,(這需要一個空分區),如果硬碟的分區已經被使用而資料你又不想要了,那就删除分區後進行下列操作。否則添加一塊新硬碟是最友善的選擇。所有storage都要進行操作。

fdisk /dev/sdb 


Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-488281249, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-488281249, default 488281249): 
Using default value 488281249


Command (m for help): p


Disk /dev/sdb: 250.0 GB, 250000000000 bytes
255 heads, 63 sectors/track, 30394 cylinders, total 488281250 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000081


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   488281249   244139601   83  Linux


Command (m for help): w
The partition table has been altered!
           

識别并挂載新的分區:

partprobe#如果不行,就重新開機一下吧。
           
mkdir -p /srv/node/sdb1
           
mkfs.xfs -i size=1024 /dev/sdb1 -f
echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
mount /srv/node/sdb1
           
chown -R swift:swift /srv/node
           

建立并配置 vim /etc/rsyncd.conf

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 10.214.0.179
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path =/srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock
           

注意:path 寫你實際的(剛才建立的)分區的位置。address修改為目前storage的位址。

編輯vim /etc/default/rsync(即設定為了開機啟動)

RSYNC_ENABLE = true
           

重新/啟動服務

service rsync restart
           

建立或檢查 vim /etc/swift/account-server.conf

[DEFAULT]
devices = /sdb1
mount_check = false  
bind_port = 6002  
user = swift
bind_ip = 0.0.0.0
workers = 2
[pipeline:main]
pipeline = account-server
[app:account-server]
use = egg:swift#account
[account-replicator]
[account-auditor]
[account-reaper]
           

建立或檢查 vim /etc/swift/container-server.conf

[DEFAULT]
devices = /sdb1
mount_check = false
bind_ip = 0.0.0.0
bind_port = 6001
workers = 2
[pipeline:main]
pipeline = container-server
[app:container-server]
use = egg:swift#container
[container-replicator]
[container-updater]
[container-auditor]
           
[container-sync]
           
注意:[container-sync]這個是添加上去的,不要忘記了
           

建立或檢查vim  /etc/swift/object-server.conf

[DEFAULT]
devices = /sdb1
mount_check = false
bind_ip = 0.0.0.0
bind_port = 6000
workers = 2
[pipeline:main]
pipeline = object-server
[app:object-server]
use = egg:swift#object
[object-replicator]
[object-updater]
[object-auditor]

           

注意去掉:object-expirer

啟動storage service:

swift-init object-server restart
swift-init object-replicator restart
swift-init object-updater restart
swift-init object-auditor restart
swift-init container-server restart
swift-init container-replicator restart
swift-init container-updater restart
swift-init container-auditor restart
swift-init account-server restart
swift-init account-replicator restart
swift-init account-auditor restart
           
也可以用指令
           
swift-init all restart
           

4.安裝和配置Proxy Node

apt-get install swift-proxy swift-doc memcached
           

 Memcached 是一個高性能的分布式記憶體對象緩存系統,用于動态Web應用以減輕資料庫負載。它通過在記憶體中緩存資料和對象來減少讀取資料庫的次數,進而提供動态、資料庫驅動網站的速度。Memcached基于一個存儲鍵/值對的hashmap。其守護程序(daemon )是用C寫的,但是用戶端可以用任何語言來編寫,并通過memcached協定與守護程序通信。

為ssl建立簽名證書

cd /etc/swift
openssl req -new -x509 -nodes -out cert.crt -keyout cert.key
           

修改/etc/memcched.conf

-l 127.0.0.1
#改為proxy節點的ip
-l  10.214.0.181
           

重新啟動memcached服務

service memcached restart
           

建立 vim /etc/swift/proxy-server.conf

[DEFAULT]
bind_port = 8080
           
bind_ip = 10.214.0.181
user = swift
           
[pipeline:main]
pipeline = catch_errors healthcheck cache authtoken keystone proxy-server
[app:proxy-server]
use = egg:swift#proxy
account_autocreate = true
[filter:keystone]
paste.filter_factory = keystone.middleware.swift_auth:filter_factory
operator_roles = admin, swiftoperator
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
service_port = 5000
service_host = 10.214.0.179
auth_port = 35357
auth_host = 10.214.0.179
auth_token =123456
admin_token =123456

auth_protocol = http
auth_uri = http://10.214.0.179:5000/
admin_tenant_name = admin
admin_user = admin
admin_password = 123456

[filter:cache]
use = egg:swift#memcache
           
memcache_servers = 10.214.0.181:11211
set log_name = cache
[filter:catch_errors]
use = egg:swift#catch_errors
[filter:healthcheck]
use = egg:swift#healthcheck
           

注意:如果你運作多個memcache ,把多個ip:端口添加到在[filter:cache]中,例如:

memcache_servers =10.1.2.3:11211,10.1.2.4:11211
           

如果不想用keystone可以用下面的配置内容代替:

[DEFAULT]
bind_port =8080
bind_ip = 10.214.0.181
user = swift

[pipeline:main]
pipeline = healthcheck cache tempauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.214.0.181:11211
           

建立環,建立account,container,object的ring,參數18代表“環”是2^18(http://blog.csdn.net/zoushidexing/article/details/7852014),參數3 代表每一個存儲對象有3份拷貝,當然這是有至少3個storage的情況下,所有的所有服務都在一個節點上,那就寫1吧。參數1代表1小時可以移動分區一次(即延遲時間)。

cd /etc/swift
swift-ring-builder account.builder create 18 3 1
swift-ring-builder container.builder create 18 3 1
swift-ring-builder object.builder create 18 3 1
           

添加ring節點,例如,我們在Zone 1中建立的一個存儲節點,ip位址為10.214.0.182,分區為/sdb1, 路徑在rsyncd.conf(存儲節點)檔案path中。100 代表裝置的權重。可以根據磁盤的容量設定對應值,比如2T的100,則1T的可以是50。端口要與配置檔案中的對應。

swift-ring-builder account.builder add z1-10.214.0.179:6002/sdb1 93
swift-ring-builder container.builder add z1-10.214.0.179:6001/sdb1 93
swift-ring-builder object.builder add z1-10.214.0.179:6000/sdb1 93
           

注意:假定有若幹個zones,每一個zones有一個node。zone應該起始于1,以1遞增。例如:swift-ring-builder account.builder add z2-10.214.0.180:6002/sdb1 28

(為了以後重建立立環境友善,可以把它們寫在vim set_devices.sh中: (一條條複制也挺煩的。)

swift-ring-builder account.builder add z1-10.214.0.179:6002/sdb1 93
swift-ring-builder container.builder add z1-10.214.0.179:6001/sdb1 93
swift-ring-builder object.builder add z1-10.214.0.179:6010/sdb1 93

swift-ring-builder account.builder add z2-10.214.0.180:6002/sdb1 28
swift-ring-builder container.builder add z2-10.214.0.180:6001/sdb1 28
swift-ring-builder object.builder add z2-10.214.0.180:6010/sdb1 28

swift-ring-builder account.builder add z3-10.214.0.182:6002/sdb1 23
swift-ring-builder container.builder add z3-10.214.0.182:6001/sdb1 23
swift-ring-builder object.builder add z3-10.214.0.182:6000/sdb1 23

swift-ring-builder account.builder
swift-ring-builder container.builder
swift-ring-builder object.builder
           
執行腳本。
           

sh set_devices.sh)

如果添加錯了而你有不止到怎麼修改,那可以删除/etc/swift下的account.builder、container.builder、 object.builder後重新建立環即重新執行swift-ring-builder account.builder create 18 3 1等。

核實ring的内容

swift-ring-builder account.builder
           
swift-ring-builder container.builder
swift-ring-builder object.builder
           

平衡rings

swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
           

成功之後會在目前目錄生成 account.ring.gz 檔案,把/etc/swift/下的account.ring.gz, container.ring.gz, and object.ring.gz拷貝到每一個proxy節點和storage節點。

确定所有的配置檔案的權限:

chown -R swift:swift /etc/swift
           

啟動proxy服務

swift-init proxy restart
           

5.重新開機服務

swift-init main restart
swift-init rest restart
           
swift-init all restart
           

Proxy節點:swift-init  proxy  start

各個Storage節點:swift-init  all  start

6.(可選)添加一個額外的proxy server

為了提高可靠性,可以添加額外的proxy server。安裝過程和上述安裝proxy node  的過程類似。但是要進行一些配置。

一旦擁有了多個proxy,就可能需要用到負載均衡。有多種負載均衡的方法可以選擇,比如:輪詢、在proxy前加一個負載均衡器,指定特定的storage。

添加proxy需要進行一些配置,當然這些配置需要需要在其proxy同時進行。

更新/etc/swift/proxy-server.conf檔案,如果使用的多個memcache servers,那麼,要如下添加IP.

[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.214.0.179:11211,10.214.0.182:11211
           

更改預設的default_cluster_url 指向負載均衡的url以代替第一次建立在/etc/swift/proxy-server.conf配置的。

[app:auth-server]
use = egg:swift#auth
default_cluster_url = https://10.214.0.180/v1
# Highly recommended to change this key to something else!
super_admin_key = devauth
           

當你改變了default_cluster_url設定,就需要删除auth database 并且重新建立openstack object storage的users,或者也可以在auth database中手動為每一個account更新正确的url。

下一步,需要拷貝ring的資訊到所有幾點,包括你建立的proxy節點。并且确定ring可以到達所有的存儲節點。

當同步了所有的節點之後,確定admin在/etc/swift有key,并且對ring檔案有充分的權限。

7.驗證設定。

使用正确的服務Identity service URL,通過export ADMINPASS=secretword導入對ADMINPASS設定。(可以通過proxy-server.conf檢視使用者和密碼)

swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 stat
           

得到an X-Storage-Url 和X-Auth-Token

curl -k -v -H 'X-Storage-User: adminUser:admin' -H 'X-Storage-Pass: $ADMINPASS' http://<AUTH_HOSTNAME>:5000/auth/v1.0
           

例如:

curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://10.214.0.181:8080/auth/v1.0
           

檢查你de賬号

curl -k -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-xstorage-url-above>
           

例如:

使用swift删除一些小檔案,命名為‘bigfile[1-2].tgz’ 給一個container 名字叫 ‘myfiles’:

$ swift -A http://<AUTH_HOSTNAME>:5000/v2.0 -U adminUser:admin -K $ADMINPASS
upload myfiles bigfile1.tgz
$ swift -A http://<AUTH_HOSTNAME>:5000/v2.0 -U adminUser:admin -K $ADMINPASS
upload myfiles bigfile2.tgz
           

使用swift從‘myfiles’ container下載下傳所有檔案

$ swift -A http://<AUTH_HOSTNAME>:5000/v2.0 -U adminUser:admin -K $ADMINPASS
download myfiles
           

8.錯誤解決提示

如果遇到問題,可以檢視日志檔案var/log/syslog

同時,在/var/log/kern.log中也會有錯誤日志。

注意端口的占用情況,我在安裝的時候由于8080端口被占用,出了好多莫名奇妙的情況。

檢視端口的指令為:

netstat -anp | grep 8080
           

如果你的服務不能夠啟動,而這個端口還被占用這,那請kill 到你用檢視端口指令看到的程序的ID号。

關聯關系:

keystone user-role-add --user 83cd1180b3794811848265cd012dffb7 --tenant_id 3804546967574be38aaa08faca23c32d --role d8c278323c9e4663bbf4ff48ebcb24d6

endpoint 配置再次提醒, 上面的service id需要換成你自己建立的service的id, 在publicurl和internalurl裡出現的AUTHxxxx,這裡的xxxx需要換成你的adminTenant的id 這裡的格式需要參考你的swift\_auth裡定義的格式, 有個reseller_prefix的設定, 這裡我們設定成AUTH。

keystone endpoint-create --region RegionOne \

                         --service_id c7a8827453ef4364af3b6809a174c6bb  \

                         --publicurl http://10.214.0.181:8080/v1/AUTH_3804546967574be38aaa08faca23c32d \

                         --adminurl http://10.214.0.181:8080 \

                         --internalurl http://10.214.0.181:8080/v1/AUTH_3804546967574be38aaa08faca23c32d

9.常用指令

檢視檔案統計結果

建立一個檔案夾

swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 post myfiles
           

myfiles 是檔案夾的名字

上傳檔案

swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 upload myfiles set_devices.sh
           

set_devices.sh 是檔案的名稱

下載下傳檔案

swift -V 2 -A http://10.214.0.179:5000/v2.0 -U service:swift -K 123456 download  myfiles set_devices.sh 
           

參考文獻:

官方文檔:http://docs.openstack.org/essex/openstack-compute/install/apt/openstack-install-guide-essex.pdf

atkisc的[Openstack] swift 安裝配置過程(all in one) :http://bbs.linuxtone.org/thread-16276-1-1.html

free_coder的Ubuntu 12.04 LTS 上安裝swift1.4.8:http://www.cnblogs.com/free--coder/archive/2012/05/28/2521135.html

cywosp的使用Swauth認證多節點安裝Swift:http://blog.csdn.net/cywosp/article/details/7428769 和 http://blog.csdn.net/cywosp/article/details/7439440

趣雲的Swift部署和動态擴充:http://blog.lightcloud.cn/?p=68#sec-4.1

Openstack Hands on lab 2: Swift安裝并使用Keystone做身份驗證:http://liangbo.me/index.php/2012/03/29/openstack-hands-on-lab-2-swift-installation-with-keystone/

繼續閱讀