Devstack. After bringing laptop back from sleep mode instance is in 'SHUTOFF' status and volume is in 'error deleting' status.

From MyWiki

(Difference between revisions)
Jump to: navigation, search
(Created page with 'After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance bac…')
 

Current revision as of 20:52, 9 January 2015

After putting my laptop to sleep, while VirtualBox was still running, I found instance in 'SHUTOFF' state and volume in 'error_deleting' state. Attempts to bring the instance back up failed and deleting instance didn't delete volume behind it.

Solution was found here

Our instance called test-instance and volume behind it is volume id 876771b2-04e6-4e80-81ba-6102deb6fa26

[stack@devstack devstack]$ nova list
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| ID                                   | Name          | Status  | Task State | Power State | Networks            |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+
| 6d7174e8-6f47-417f-bd10-b163e77e504c | test-instance | SHUTOFF | -          | Shutdown    | private=172.16.20.5 |
+--------------------------------------+---------------+---------+------------+-------------+---------------------+

[stack@devstack devstack]$ openstack volume list
+--------------------------------------+--------------+----------------+------+----------+
| ID                                   | Display Name | Status         | Size | Attached |
+--------------------------------------+--------------+----------------+------+----------+
| 876771b2-04e6-4e80-81ba-6102deb6fa26 |              | error          |    1 |          |
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e |              | error_deleting |    1 |          |
+--------------------------------------+--------------+----------------+------+----------+

[stack@devstack devstack]$ nova show test-instance
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | devstack                                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | devstack                                                 |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                        |
| OS-EXT-STS:power_state               | 4                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | stopped                                                  |
| OS-SRV-USG:launched_at               | 2014-12-24T03:38:09.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2014-12-24T03:37:53Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 47c2aed6766c945e2da09de617a556e11c33b00a841efe70ed4fdd56 |
| id                                   | 6d7174e8-6f47-417f-bd10-b163e77e504c                     |
| image                                | Attempt to boot from volume - no image supplied          |
| key_name                             | devstack-ssh                                             |
| metadata                             | {}                                                       |
| name                                 | test-instance                                            |
| os-extended-volumes:volumes_attached | [{"id": "876771b2-04e6-4e80-81ba-6102deb6fa26"}]         |
| private network                      | 172.16.20.5                                              |
| security_groups                      | default, ssh_icmp                                        |
| status                               | SHUTOFF                                                  |
| tenant_id                            | 56f102d9b8704e78bc13d80c92463eb5                         |
| updated                              | 2015-01-08T19:22:20Z                                     |
| user_id                              | 20cf3d728c444f6880fb1d554b9067f5                         |
+--------------------------------------+----------------------------------------------------------+

[stack@devstack devstack]$ openstack volume show 876771b2-04e6-4e80-81ba-6102deb6fa26
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                          | Value                                                                                                                                                                                                                                                                                                                                                                              |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments                    | [{u'device': u'vda', u'server_id': u'6d7174e8-6f47-417f-bd10-b163e77e504c', u'id': u'876771b2-04e6-4e80-81ba-6102deb6fa26', u'host_name': None, u'volume_id': u'876771b2-04e6-4e80-81ba-6102deb6fa26'}]                                                                                                                                                                            |
| availability_zone              | nova                                                                                                                                                                                                                                                                                                                                                                               |
| bootable                       | true                                                                                                                                                                                                                                                                                                                                                                               |
| created_at                     | 2014-12-24T03:37:55.000000                                                                                                                                                                                                                                                                                                                                                         |
| display_description            |                                                                                                                                                                                                                                                                                                                                                                                    |
| display_name                   |                                                                                                                                                                                                                                                                                                                                                                                    |
| encrypted                      | False                                                                                                                                                                                                                                                                                                                                                                              |
| id                             | 876771b2-04e6-4e80-81ba-6102deb6fa26                                                                                                                                                                                                                                                                                                                                               |
| os-vol-host-attr:host          | devstack                                                                                                                                                                                                                                                                                                                                                                           |
| os-vol-mig-status-attr:migstat | None                                                                                                                                                                                                                                                                                                                                                                               |
| os-vol-mig-status-attr:name_id | None                                                                                                                                                                                                                                                                                                                                                                               |
| project_id                     | 56f102d9b8704e78bc13d80c92463eb5                                                                                                                                                                                                                                                                                                                                                   |
| properties                     | attached_mode='rw', readonly='False'                                                                                                                                                                                                                                                                                                                                               |
| size                           | 1                                                                                                                                                                                                                                                                                                                                                                                  |
| snapshot_id                    | None                                                                                                                                                                                                                                                                                                                                                                               |
| source_volid                   | None                                                                                                                                                                                                                                                                                                                                                                               |
| status                         | error                                                                                                                                                                                                                                                                                                                                                                              |
| type                           | None                                                                                                                                                                                                                                                                                                                                                                               |
| volume_image_metadata          | {u'kernel_id': u'20ee87ad-94b4-4cab-9d58-5959b7f51412', u'container_format': u'ami', u'min_ram': u'0', u'ramdisk_id': u'9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf', u'disk_format': u'ami', u'image_name': u'cirros-0.3.2-x86_64-uec', u'image_id': u'efdf518c-c021-47f6-94b1-fe93b0fa3969', u'checksum': u'4eada48c2843d2a262c814ddc92ecf2c', u'min_disk': u'0', u'size': u'25165824'} |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Volume 853e0bcc-05cf-484d-859d-1bcbfa46a55e was left over from the similar situation - sleep mode resulted in an instance left in limbo. Attempt to delete it fails:

[stack@devstack devstack]$ openstack volume delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e
ERROR: openstack Invalid volume: Volume status must be available or error, but current status is: error_deleting (HTTP 400) (Request-ID: req-6e1f56ce-3805-41a9-bb68-d16de9dc35f7)

Let's try to delete the SHUTOFF instance:

[stack@devstack devstack]$ nova delete test-instance
Request to delete server test-instance has been accepted.

[stack@devstack devstack]$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

Instance is gone, but not its volume 876771b2-04e6-4e80-81ba-6102deb6fa26, which is now also in 'error_deleting' state:

[stack@devstack devstack]$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting |              |  1   |     None    |   true   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

Check that we don't have any snapshots before we attempt anything else:

[stack@devstack devstack]$ cinder snapshot-list
+----+-----------+--------+--------------+------+
| ID | Volume ID | Status | Display Name | Size |
+----+-----------+--------+--------------+------+
+----+-----------+--------+--------------+------+

Let's check now what iSCSI targets do we have:

[stack@devstack devstack]$ sudo tgt-admin -s
Target 1: iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
    Account information:
    ACL information:
        ALL

So, we have only one target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and no target for volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e Trying to delete the target for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26 and it works fine:

[stack@devstack devstack]$ sudo tgt-admin -delete iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26
[stack@devstack devstack]$ sudo tgt-admin -s

But cinder force-delete is not doing the magic:

[stack@devstack devstack]$ cinder force-delete 876771b2-04e6-4e80-81ba-6102deb6fa26
[stack@devstack devstack]$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting |              |  1   |     None    |   true   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

[stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e
[stack@devstack devstack]$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | error_deleting |              |  1   |     None    |   true   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

OK. Let's try to reset-state and see if it helps:

[stack@devstack devstack]$ cinder reset-state 876771b2-04e6-4e80-81ba-6102deb6fa26
[stack@devstack devstack]$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 |   available    |              |  1   |     None    |   true   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

[stack@devstack devstack]$ cinder reset-state 853e0bcc-05cf-484d-859d-1bcbfa46a55e
[stack@devstack devstack]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | available |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 | available |              |  1   |     None    |   true   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[stack@devstack devstack]$ cinder force-delete 853e0bcc-05cf-484d-859d-1bcbfa46a55e
[stack@devstack devstack]$ cinder list
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 853e0bcc-05cf-484d-859d-1bcbfa46a55e | error_deleting |              |  1   |     None    |   true   |             |
| 876771b2-04e6-4e80-81ba-6102deb6fa26 |   available    |              |  1   |     None    |   true   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

Still nothing. Back to the same 'error_deleting' status.

Time for the tough measures - delete the records in MySQL volumes table:

[stack@devstack devstack]$ mysql -u root cinder
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 248
Server version: 5.1.73-log Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from volumes;
+---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+
| created_at          | updated_at          | deleted_at | deleted | id                                   | ec2_id | user_id                          | project_id                       | host     | size | availability_zone | instance_uuid | mountpoint | attach_time | status         | attach_status | scheduled_at        | launched_at         | terminated_at       | display_name | display_description | provider_location                                                                           | provider_auth                                  | snapshot_id | volume_type_id | source_volid | bootable | attached_host | provider_geometry | _name_id | encryption_key_id | migration_status |
+---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+
| 2014-12-23 20:13:28 | 2014-12-24 03:34:07 | NULL       |       0 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL   | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack |    1 | nova              | NULL          | NULL       | NULL        | error_deleting | detached      | 2014-12-23 20:13:29 | 2014-12-23 20:13:31 | 2014-12-24 03:34:06 |              |                     | 172.16.10.10:3260,1 iqn.2010-10.org.openstack:volume-853e0bcc-05cf-484d-859d-1bcbfa46a55e 1 | CHAP Rt5PCuZJFwahV76e74u2 S6e7A2tSGcwyKvdd4PG5 | NULL        | NULL           | NULL         |        1 | NULL          | NULL              | NULL     | NULL              | NULL             |
| 2014-12-24 03:37:55 | 2015-01-09 08:58:15 | NULL       |       0 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL   | 20cf3d728c444f6880fb1d554b9067f5 | 56f102d9b8704e78bc13d80c92463eb5 | devstack |    1 | nova              | NULL          | NULL       | NULL        | error_deleting | detached      | 2014-12-24 03:37:55 | 2014-12-24 03:37:57 | 2015-01-09 08:58:15 |              |                     | 172.16.10.10:3260,2 iqn.2010-10.org.openstack:volume-876771b2-04e6-4e80-81ba-6102deb6fa26 1 | CHAP 5mgeSdMVPxTCPMFhqc2R wv4NJM9tVLKhWpZwpaB2 | NULL        | NULL           | NULL         |        1 | NULL          | NULL              | NULL     | NULL              | NULL             |
+---------------------+---------------------+------------+---------+--------------------------------------+--------+----------------------------------+----------------------------------+----------+------+-------------------+---------------+------------+-------------+----------------+---------------+---------------------+---------------------+---------------------+--------------+---------------------+---------------------------------------------------------------------------------------------+------------------------------------------------+-------------+----------------+--------------+----------+---------------+-------------------+----------+-------------------+------------------+
2 rows in set (0.00 sec)

Attempt to delete volume id 853e0bcc-05cf-484d-859d-1bcbfa46a55e fails due to foreign key constraint:

mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e';
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_glance_metadata`, CONSTRAINT `volume_glance_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))

Check what we have in volume_glance_metadata:

mysql> select * from volume_glance_metadata;
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
| created_at          | updated_at | deleted_at | deleted | id | volume_id                            | snapshot_id | key              | value                                |
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | kernel_id        | 20ee87ad-94b4-4cab-9d58-5959b7f51412 |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | container_format | ami                                  |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  3 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | min_ram          | 0                                    |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  4 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | ramdisk_id       | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  5 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | disk_format      | ami                                  |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  6 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | image_name       | cirros-0.3.2-x86_64-uec              |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  7 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | image_id         | efdf518c-c021-47f6-94b1-fe93b0fa3969 |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  8 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | checksum         | 4eada48c2843d2a262c814ddc92ecf2c     |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 |  9 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | min_disk         | 0                                    |
| 2014-12-23 20:13:31 | NULL       | NULL       |       0 | 10 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | NULL        | size             | 25165824                             |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | kernel_id        | 20ee87ad-94b4-4cab-9d58-5959b7f51412 |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | container_format | ami                                  |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | min_ram          | 0                                    |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | ramdisk_id       | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | disk_format      | ami                                  |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | image_name       | cirros-0.3.2-x86_64-uec              |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | image_id         | efdf518c-c021-47f6-94b1-fe93b0fa3969 |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | checksum         | 4eada48c2843d2a262c814ddc92ecf2c     |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | min_disk         | 0                                    |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | size             | 25165824                             |
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
20 rows in set (0.00 sec)

Clean volume_glance_metadata now and try to delete the volume from volumes table again:

mysql> delete from volume_glance_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e';
Query OK, 10 rows affected (0.00 sec)

mysql> select * from volume_glance_metadata;
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
| created_at          | updated_at | deleted_at | deleted | id | volume_id                            | snapshot_id | key              | value                                |
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 11 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | kernel_id        | 20ee87ad-94b4-4cab-9d58-5959b7f51412 |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 12 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | container_format | ami                                  |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 13 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | min_ram          | 0                                    |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 14 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | ramdisk_id       | 9b59b15c-1a65-42e3-8fe3-a2c8caf1a0cf |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 15 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | disk_format      | ami                                  |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 16 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | image_name       | cirros-0.3.2-x86_64-uec              |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 17 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | image_id         | efdf518c-c021-47f6-94b1-fe93b0fa3969 |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 18 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | checksum         | 4eada48c2843d2a262c814ddc92ecf2c     |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 19 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | min_disk         | 0                                    |
| 2014-12-24 03:37:57 | NULL       | NULL       |       0 | 20 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | NULL        | size             | 25165824                             |
+---------------------+------------+------------+---------+----+--------------------------------------+-------------+------------------+--------------------------------------+
10 rows in set (0.00 sec)

mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e';
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))

Boom! We have more in volume_admin_metadata. Clean it too:

mysql> select * from volume_admin_metadata;
+---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+
| created_at          | updated_at | deleted_at          | deleted | id | volume_id                            | key           | value |
+---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+
| 2014-12-23 20:13:34 | NULL       | NULL                |       0 |  1 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | readonly      | False |
| 2014-12-23 20:13:34 | NULL       | 2014-12-24 03:34:06 |       1 |  2 | 853e0bcc-05cf-484d-859d-1bcbfa46a55e | attached_mode | rw    |
| 2014-12-24 03:37:59 | NULL       | NULL                |       0 |  3 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | readonly      | False |
| 2014-12-24 03:38:00 | NULL       | 2015-01-09 08:58:14 |       1 |  4 | 876771b2-04e6-4e80-81ba-6102deb6fa26 | attached_mode | rw    |
+---------------------+------------+---------------------+---------+----+--------------------------------------+---------------+-------+
4 rows in set (0.00 sec)

mysql> delete from volume_admin_metadata where volume_id='853e0bcc-05cf-484d-859d-1bcbfa46a55e';
Query OK, 2 rows affected (0.00 sec)

Now again back to cleaning volumes table and it works this time:

mysql> delete from volumes where id='853e0bcc-05cf-484d-859d-1bcbfa46a55e';
Query OK, 1 row affected (0.00 sec)

Repeat the process for volume id 876771b2-04e6-4e80-81ba-6102deb6fa26:

mysql> delete from volume_admin_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26';
Query OK, 2 rows affected (0.00 sec)

mysql> delete from volume_glance_metadata where volume_id='876771b2-04e6-4e80-81ba-6102deb6fa26';
Query OK, 10 rows affected (0.01 sec)

mysql> delete from volumes where id='876771b2-04e6-4e80-81ba-6102deb6fa26';
Query OK, 1 row affected (0.00 sec)

mysql> \q
Bye

And it's all clean now:

[stack@devstack devstack]$ cinder list
+----+--------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+----+--------+--------------+------+-------------+----------+-------------+
+----+--------+--------------+------+-------------+----------+-------------+
Personal tools