存儲的分類：
塊存儲：
硬碟
LVM
DAS(Direct Attach Storage)
是直接連接配接于主機伺服器的一種存儲方式，每一台主機伺服器有獨立的儲存設備，每台主機伺服器的儲存設備無法互通。
通常用單一網絡環境下且資料交換量不大，性能要求不高的環境下，可以說是一種應用較早的技術實作
SAN
1、FC-SAN
2、IP-SAN
分布式存儲-Ceph
Ceph是開源實作的PB級别的分布式檔案系統，其分布式對象存儲機制為上層提供了檔案接口、塊存儲接口和對象存儲接口
DAS(Direct Attach Storage)
檔案存儲：
FS
對象存儲：
SAN(storage Area Network)
NAS
Cinder元件：
chnder-api。接受API請求并将請求路由到cinder-volume來執行。
cinder-volume。相應請求，讀取或寫向塊存儲資料庫為維護狀态，通過資訊隊列機制與其他程序互動（如cinder-scheduler），或直接與上層塊存儲提供的硬體或軟體進行互動。通過驅動結構，他可以與衆多的存儲提供者進行互動。
cinder-scheduler。守護程序。類型于nova-scheduler，為存儲卷的執行個體選取最優的塊存儲供應節點。
沒有Cinder時候虛拟機磁盤存放位置；
[[email protected] linux-node2 ~]# ls /var/lib/nova/instances/
_base compute_nodes e7585ece-c8e9-4ad8-8796-8a58bca459bf locks snapshots
[[email protected] linux-node2 ~]# ls /var/lib/nova/instances/e7585ece-c8e9-4ad8-8796-8a58bca459bf/
console.log disk disk.info libvirt.xml
安裝Cinder控制節點：
[[email protected] ~]# yum -y install openstack-cinder python-cinderclient
檢視是否已建立cinder資料庫，我這裡在keystone安裝時候已經建立；
[[email protected] ~]# mysql -uroot -psmoke520
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 264
Server version: 10.1.20-MariaDB MariaDB Server
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| cinder |
| glance |
| information_schema |
| keystone |
| mysql |
| neutron |
| nova |
| performance_schema |
+--------------------+
8 rows in set (1.25 sec)
MariaDB [(none)]> exit;
Bye
配置Cinder：
[[email protected] ~]# vim /etc/cinder/cinder.conf
[database]
connection = mysql://cinder:[email protected]/cinder
同步資料庫：
[[email protected] ~]# su -s /bin/sh -c "cinder-manage db sync" cinder
[[email protected] ~]# mysql -ucinder -pcinder -h192.168.56.11 -e "use cinder;show tables;"
+----------------------------+
| Tables_in_cinder |
+----------------------------+
| backups |
| cgsnapshots |
| consistencygroups |
| driver_initiator_data |
| encryption |
| image_volume_cache_entries |
| iscsi_targets |
| migrate_version |
| quality_of_service_specs |
| quota_classes |
| quota_usages |
| quotas |
| reservations |
| services |
| snapshot_metadata |
| snapshots |
| transfers |
| volume_admin_metadata |
| volume_attachment |
| volume_glance_metadata |
| volume_metadata |
| volume_type_extra_specs |
| volume_type_projects |
| volume_types |
| volumes |
+----------------------------+
建立使用者：
[[email protected] ~]# source admin-openrc.sh
使用者名cinder，密碼cinder；
[[email protected] ~]# openstack user create --domain default --password-prompt cinder
User Password:
Repeat User Password:
+-----------+----------------------------------+
| Field | Value |
+-----------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 5ffa5acdd3d8438e994a6fcd9319708a |
| name | cinder |
+-----------+----------------------------------+
将cinder使用者加入service項目并賦予admin角色；
[[email protected] ~]# openstack role add --project service --user cinder admin
更改chinder配置檔案：
[[email protected] ~]# vim /etc/cinder/cinder.conf
[DEFAULT]
auth_strategy = keystone
rpc_backend = rabbit
glance_host = 192.168.56.11
[keystone_authtoken]
auth_uri = http://192.168.56.11:5000
auth_url = http://192.168.56.11:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder
[oslo_messaging_rabbit]
rabbit_host = 192.168.56.11
rabbit_port = 5672
rabbit_userid = openstack
rabbit_password = openstack
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[[email protected] ~]# grep '^[a-z]' /etc/cinder/cinder.conf
glance_host = 192.168.56.11
auth_strategy = keystone
rpc_backend = rabbit
connection = mysql://cinder:[email protected]/cinder
auth_uri = http://192.168.56.11:5000
auth_url = http://192.168.56.11:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder
lock_path = /var/lib/cinder/tmp
rabbit_host = 192.168.56.11
rabbit_port = 5672
rabbit_userid = openstack
rabbit_password = openstack
[[email protected] ~]# vim /etc/nova/nova.conf
[cinder]
os_region_name = RegionOne
重新開機服務：
[[email protected] ~]# systemctl restart openstack-nova-api.service
[[email protected] ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-api.service to /usr/lib/systemd/system/openstack-cinder-api.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-scheduler.service to /usr/lib/systemd/system/openstack-cinder-scheduler.service.
[[email protected] ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service
注冊Cinder：
建立Cinder v1服務：
[[email protected] ~]# openstack service create --name cinder \
--description "OpenStack Block Storage" volume
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 44c68286553947049187cd9886a85807 |
| name | cinder |
| type | volume |
+-------------+----------------------------------+
建立Cinder v2服務：
[[email protected] ~]# openstack service create --name cinderv2 \
--description "OpenStack Block Storage" volumev2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Block Storage |
| enabled | True |
| id | 9bc7d84ec806418d9046635275518741 |
| name | cinderv2 |
| type | volumev2 |
+-------------+----------------------------------+
建立v1 endpoint：
公網注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne \
> volume public http://192.168.56.11:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 22d5172503014abc99e1448e8990fe01 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 44c68286553947049187cd9886a85807 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.56.11:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+
私網注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne volume internal http://192.168.56.11:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 184ede7d6cb64e52a14e884cd90b8455 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 44c68286553947049187cd9886a85807 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.56.11:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+
admin注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne volume admin http://192.168.56.11:8776/v1/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | 38796e145c55401e8140815af281d0fd |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 44c68286553947049187cd9886a85807 |
| service_name | cinder |
| service_type | volume |
| url | http://192.168.56.11:8776/v1/%(tenant_id)s |
+--------------+--------------------------------------------+
建立v2 endpoint：
公網注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.56.11:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | c7246b6f7c8d4227b2cb0b03c7ffe694 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 9bc7d84ec806418d9046635275518741 |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.56.11:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+
私網注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.56.11:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | e738d99d2a7a46e5a122ed6b006de63e |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 9bc7d84ec806418d9046635275518741 |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.56.11:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+
admin注冊：
[[email protected] ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.56.11:8776/v2/%\(tenant_id\)s
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| enabled | True |
| id | c7c6845176214a64ab9abefb7cc7acf8 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 9bc7d84ec806418d9046635275518741 |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | http://192.168.56.11:8776/v2/%(tenant_id)s |
+--------------+--------------------------------------------+
安裝Cinder存儲節點到計算節點：
需要先關機給計算節點添加硬碟給Cinder使用；
[[email protected] linux-node2 ~]# fdisk -l
磁盤 /dev/sda：53.7 GB, 53687091200 位元組，104857600 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/實體)：512 位元組 / 512 位元組
I/O 大小(最小/最佳)：512 位元組 / 512 位元組
磁盤标簽類型：dos
磁盤辨別符：0x000c55d1
裝置 Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 104857599 51379200 8e Linux LVM
磁盤 /dev/sdb：53.7 GB, 53687091200 位元組，104857600 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/實體)：512 位元組 / 512 位元組
I/O 大小(最小/最佳)：512 位元組 / 512 位元組
磁盤 /dev/mapper/cl-root：50.5 GB, 50457477120 位元組，98549760 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/實體)：512 位元組 / 512 位元組
I/O 大小(最小/最佳)：512 位元組 / 512 位元組
磁盤 /dev/mapper/cl-swap：2147 MB, 2147483648 位元組，4194304 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/實體)：512 位元組 / 512 位元組
I/O 大小(最小/最佳)：512 位元組 / 512 位元組
建立LVM：
[[email protected] linux-node2 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
[[email protected] linux-node2 ~]# vgcreate cinder-volumes /dev/sdb
Volume group "cinder-volumes" successfully created
設定LVM配置檔案：
[[email protected] linux-node2 ~]# vim /etc/lvm/lvm.conf
filter = [ "a/sdb/", "r/.*/" ]
[[email protected] linux-node2 ~]# yum -y install openstack-cinder targetcli python-oslo-policy
配置Cinder：
[[email protected] linux-node2 ~]# scp [email protected]:/etc/cinder/cinder.conf /etc/cinder/
[email protected]'s password:
cinder.conf 100% 117KB 116.7KB/s 00:00
[[email protected] linux-node2 ~]# vim /etc/cinder/cinder.conf
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = lioadm
[DEFAULT]
enabled_backends = lvm
[[email protected] linux-node2 ~]# grep '^[a-z]' /etc/cinder/cinder.conf
glance_host = 192.168.56.11
auth_strategy = keystone
enabled_backends = lvm
rpc_backend = rabbit
connection = mysql://cinder:[email protected]/cinder
auth_uri = http://192.168.56.11:5000
auth_url = http://192.168.56.11:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder
lock_path = /var/lib/cinder/tmp
rabbit_host = 192.168.56.11
rabbit_port = 5672
rabbit_userid = openstack
rabbit_password = openstack
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = lioadm
啟動Cinder：
[[email protected] linux-node2 ~]# systemctl enable openstack-cinder-volume.service target.service
Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[[email protected] linux-node2 ~]# systemctl start openstack-cinder-volume.service target.service
如果啟動openstack-cinder-volume.service報錯為ImportError: No module named Crypto.Random，主要是因為缺少pycrypto子產品，通過pip install pycryton安裝；
[[email protected] ~]# source admin-openrc.sh
[[email protected] ~]# cinder service-list
+------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | hostnamelinux-node1.smoke.com | nova | enabled | up | 2018-10-13T10:24:53.000000 | - |
| cinder-volume | hostname [email protected] | nova | enabled | up | 2018-10-13T10:24:29.000000 | - |
+------------------+------------------------------------+------+---------+-------+----------------------------+-----------------+
建立雲硬碟：
使用demo密碼demo使用者登入Dashboard；
建立成功；
建立一個雲硬碟會建立一個LVS；
[[email protected] linux-node2 ~]# lvdisplay
--- Logical volume ---
LV Path /dev/cinder-volumes/volume-4d5fbaae-8eaa-43ea-aff7-933b5a1f7564
LV Name volume-4d5fbaae-8eaa-43ea-aff7-933b5a1f7564
VG Name cinder-volumes
LV UUID 1l2TMz-ZjRM-R3Kw-as7G-Ga7V-ufFG-ePfq17
LV Write Access read/write
LV Creation host, time hostname linux-node2.smoke.com, 2018-10-13 18:35:48 +0800
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:2
挂載雲硬碟：
選挂載的虛拟機執行個體；
挂載成功；
連接配接虛拟機執行個體進行驗證；
$ sudo fdisk -l
Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/vda1 * 16065 2088449 1036192+ 83 Linux
Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/vdb doesn't contain a valid partition table
$ sudo fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x12712ffb.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):
Using default value 2097151
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
$ sudo mkfs.ext4 /dev/vdb1
mke2fs 1.42.2 (27-Mar-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 261888 blocks
13094 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkdir /data
$ sudo mount /dev/vdb1 /data/
$ sudo mount
rootfs on / type rootfs (rw)
/dev on /dev type devtmpfs (rw,relatime,size=248148k,nr_inodes=62037,mode=755)
/dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered)
/proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755)
/dev/vdb1 on /data type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
$ cd /data/
$ ls
lost+found
$ sudo mkdir hehe
$ sudo touch test
$ ls
hehe lost+found test
解除安裝雲硬碟，不能直接删除硬碟，虛拟機會壞掉；
$ cd
$ sudo umount /data
$ sudo mount
rootfs on / type rootfs (rw)
/dev on /dev type devtmpfs (rw,relatime,size=248148k,nr_inodes=62037,mode=755)
/dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered)
/proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755)
建立執行個體允許修改密碼（控制節點）：
[[email protected] ~]# vim /etc/openstack-dashboard/local_settings
OPENSTACK_HYPERVISOR_FEATURES = {
'can_set_mount_point': True,
'can_set_password': True,
'requires_keypair': True,
}
[[email protected] ~]# systemctl restart httpd
允許注入密碼（計算節點）：
[[email protected] linux-node2 ~]# vim /etc/nova/nova.conf
[libvirt]
inject_password=true
inject_key=true
[[email protected] linux-node2 ~]# systemctl restart openstack-nova-compute
建立虛拟機：
詳情：
通路和安全；
建立後腳本：
建立成功；
使用設定的密碼smoke520進行登入；
轉載于:https://blog.51cto.com/smoke520/2299010