Skip to content

Commit ad9a4c1

Browse files
author
Brian Curtin
committed
Implement Swift Proxy object and example
This implements the proxy that is exposed as Connection.object_store, and includes some example usage as well as documentation and a user guide. python -m examples.object_store --list-containers python -m examples.object_store --list-objects <container> python -m examples.object_store --upload-directory <directory> --pattern <glob pattern> Ex: ... --upload-directory pictures/ --pattern "*.jpg" Change-Id: I8739ebca2ac77ea4a4d6f4e3ff30a3a253d8b636
1 parent a250d88 commit ad9a4c1

File tree

9 files changed

+993
-2
lines changed

9 files changed

+993
-2
lines changed

‎doc/source/code/connection.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from openstack import connection
2+
conn = connection.Connection(auth_url="http://openstack:5000/v3",
3+
project_name="big_project",
4+
user_name="SDK_user",
5+
password="Super5ecretPassw0rd")
6+

‎doc/source/highlevel/object_store.rst‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Object Store API
2+
================
3+
4+
For details on how to use this API, see :doc:`/userguides/object_store`
5+
6+
.. automodule:: openstack.object_store.v1._proxy
7+
8+
The Object Store Class
9+
----------------------
10+
11+
The Object Store high-level interface is exposed as the ``object_store``
12+
object on :class:`~openstack.connection.Connection` objects.
13+
14+
.. autoclass:: openstack.object_store.v1._proxy.Proxy
15+
:members:

‎doc/source/index.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,22 @@ Welcome!
1111
contributing
1212
glossary
1313

14+
User Guides
15+
-----------
16+
17+
.. toctree::
18+
:maxdepth: 1
19+
20+
userguides/object_store
21+
22+
High-Level Interface
23+
--------------------
24+
25+
.. toctree::
26+
:maxdepth: 1
27+
28+
highlevel/object_store
29+
1430
Resource Level Classes
1531
----------------------
1632

‎doc/source/userguides/object_store.rst‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
Using the OpenStack Object Store API
2+
====================================
3+
4+
The Object Store API operates on two things: containers and objects.
5+
6+
Before working with the ``object_store`` API, you'll need to obtain a
7+
:class:`~openstack.connection.Connection` object like so.
8+
9+
.. literalinclude:: /code/connection.py
10+
11+
Working with Containers
12+
-----------------------
13+
14+
Listing Containers
15+
******************
16+
17+
To list existing containers, use the
18+
:meth:`~openstack.object_store.v1._proxy.Proxy.containers` method. ::
19+
20+
>>> for cont in conn.object_store.containers():
21+
... print cont
22+
...
23+
Container: {u'count': 5, u'bytes': 500, u'name': u'my container'}
24+
Container: {u'count': 0, u'bytes': 0, u'name': u'empty container'}
25+
Container: {u'count': 100, u'bytes': 1000000, u'name': u'another container'}
26+
27+
The ``containers`` method returns a generator which yields
28+
:class:`~openstack.object_store.v1.container.Container` objects. It handles
29+
pagination for you, which can be adjusted via the ``limit`` argument.
30+
By default, the ``containers`` method will yield as many containers as the
31+
service will return, and it will continue requesting until it receives
32+
no more. ::
33+
34+
>>> for cont in conn.object_store.containers(limit=500):
35+
... print(cont)
36+
...
37+
<500 Containers>
38+
... another request transparently made to the Object Store service
39+
<500 more Containers>
40+
...
41+
42+
Creating Containers
43+
*******************
44+
45+
To create a container, use the
46+
:meth:`~openstack.object_store.v1._proxy.Proxy.create_container` method. ::
47+
48+
>>> cont = conn.object_store.create_container("new container".decode("utf8"))
49+
>>> cont
50+
Container: {'name': u'new container'}
51+
52+
You can also create containers by passing in a
53+
:class:`~openstack.object_store.v1.container.Container` resource. This is
54+
helpful if you wanted to create another container which uses the same metadata
55+
settings that another container has. ::
56+
57+
>>> from copy import copy
58+
>>> print cont.name, cont.read_ACL
59+
MyContainer .r:mysite.com
60+
>>> new_cont = copy(cont)
61+
>>> new_cont.name = "copied container"
62+
>>> conn.object_store.create_container(new_cont)
63+
Container: {u'name': 'copied container', 'x-container-read': '.r:mysite.com'}
64+
65+
Working with Container Metadata
66+
*******************************
67+
68+
To get the metadata for a container, use the
69+
:meth:`~openstack.object_store.v1._proxy.Proxy.get_container_metadata` method.
70+
This method either takes the name of a container, or a
71+
:class:`~openstack.object_store.v1.container.Container` object, and it returns
72+
a `Container` object with all of its metadata attributes set. ::
73+
74+
>>> cont = conn.object_store.get_container_metadata("new container".decode("utf8"))
75+
Container: {'content-length': '0', 'x-container-object-count': '0',
76+
'name': u'new container', 'accept-ranges': 'bytes',
77+
'x-trans-id': 'tx22c5de63466e4c05bb104-0054740c39',
78+
'date': 'Tue, 25 Nov 2014 04:57:29 GMT',
79+
'x-timestamp': '1416889793.23520', 'x-container-read': '.r:mysite.com',
80+
'x-container-bytes-used': '0', 'content-type': 'text/plain; charset=utf-8'}
81+
82+
To set the metadata for a container, use the
83+
:meth:`~openstack.object_store.v1._proxy.Proxy.set_container_metadata` method.
84+
This method takes a :class:`~openstack.object_store.v1.container.Container`
85+
object. For example, to grant another user write access to this container,
86+
you can set the
87+
:attr:`~openstack.object_store.v1.container.Container.write_ACL` on a
88+
resource and pass it to `set_container_metadata`. ::
89+
90+
>>> cont.write_ACL = "big_project:another_user"
91+
>>> conn.object_store.set_container_metadata(cont)
92+
Container: {'content-length': '0', 'x-container-object-count': '0',
93+
'name': u'my new container', 'accept-ranges': 'bytes',
94+
'x-trans-id': 'txc3ee751f971d41de9e9f4-0054740ec1',
95+
'date': 'Tue, 25 Nov 2014 05:08:17 GMT',
96+
'x-timestamp': '1416889793.23520', 'x-container-read': '.r:mysite.com',
97+
'x-container-bytes-used': '0', 'content-type': 'text/plain; charset=utf-8',
98+
'x-container-write': 'big_project:another_user'}
99+
100+
Working with Objects
101+
--------------------
102+
103+
Objects are held in containers. From an API standpoint, you work with
104+
them using similarly named methods, typically with an additional argument
105+
to specify their container.
106+
107+
Listing Objects
108+
***************
109+
110+
To list the objects that exist in a container, use the
111+
:meth:`~openstack.object_store.v1._proxy.Proxy.objects` method.
112+
113+
If you have a :class:`~openstack.object_store.v1.container.Container`
114+
object, you can pass it to ``objects``. ::
115+
116+
>>> print cont.name
117+
pictures
118+
>>> for obj in conn.object_store.objects(cont):
119+
... print obj
120+
...
121+
Object: {u'hash': u'0522d4ccdf9956badcb15c4087a0c4cb',
122+
u'name': u'pictures/selfie.jpg', u'bytes': 15744,
123+
'last-modified': u'2014-10-31T06:33:36.618640',
124+
u'last_modified': u'2014-10-31T06:33:36.618640',
125+
u'content_type': u'image/jpeg', 'container': u'pictures',
126+
'content-type': u'image/jpeg'}
127+
...
128+
129+
Similar to the :meth:`~openstack.object_store.v1._proxy.Proxy.containers`
130+
method, ``objects`` returns a generator which yields
131+
:class:`~openstack.object_store.v1.obj.Object` objects stored in the
132+
container. It also handles pagination for you, which you can adjust
133+
with the ``limit`` parameter, otherwise making each request for the maximum
134+
that your Object Store will return.
135+
136+
If you have the name of a container instead of an object, you can also
137+
pass that to the ``objects`` method. ::
138+
139+
>>> for obj in conn.object_store.objects("pictures".decode("utf8"),
140+
limit=100):
141+
... print obj
142+
...
143+
<100 Objects>
144+
... another request transparently made to the Object Store service
145+
<100 more Objects>
146+
147+
Getting Object Data
148+
*******************
149+
150+
Once you have an :class:`~openstack.object_store.v1.obj.Object`, you get
151+
the data stored inside of it with the
152+
:meth:`~openstack.object_store.v1._proxy.Proxy.get_object_data` method. ::
153+
154+
>>> print ob.name
155+
message.txt
156+
>>> data = conn.object_store.get_object_data(ob)
157+
>>> print data
158+
Hello, world!
159+
160+
Additionally, if you want to save the object to disk, the
161+
:meth:`~openstack.object_store.v1._proxy.Proxy.save_object` convenience
162+
method takes an :class:`~openstack.object_store.v1.obj.Object` and a
163+
``path`` to write the contents to. ::
164+
165+
>>> conn.object_store.save_object(ob, "the_message.txt")
166+
167+
Creating Objects
168+
****************
169+
170+
Once you have data you'd like to store in the Object Store service, you use
171+
the :meth:`~openstack.object_store.v1._proxy.Proxy.create_object` method.
172+
This method takes the ``data`` to be stored, along with an ``obj`` and
173+
``container``. The ``obj`` can either be the name of an object or an
174+
:class:`~openstack.object_store.v1.obj.Object` instance, and ``container``
175+
can either be the name of a container or an
176+
:class:`~openstack.object_store.v1.container.Container` instance. ::
177+
178+
>>> hello = conn.object_store.create_object("Hello, world!",
179+
"helloworld.txt".decode("utf8"),
180+
"My Container".decode("utf8"))
181+
>>> print hello
182+
Object: {'content-length': '0', 'container': u'My Container',
183+
'name': u'helloworld.txt',
184+
'last-modified': 'Tue, 25 Nov 2014 17:39:29 GMT',
185+
'etag': '5eb63bbbe01eeed093cb22bb8f5acdc3',
186+
'x-trans-id': 'tx3035d41b03334aeaaf3dd-005474bed0',
187+
'date': 'Tue, 25 Nov 2014 17:39:28 GMT',
188+
'content-type': 'text/html; charset=UTF-8'}
189+
190+
If you have an existing object and want to update its data, you can easily
191+
do that by passing new ``data`` along with existing
192+
:class:`~openstack.object_store.v1.obj.Object` and
193+
:class:`~openstack.object_store.v1.container.Container` instances. ::
194+
195+
>>> conn.object_store.create_object("Hola, mundo!", hello, cont)
196+
197+
Working with Object Metadata
198+
****************************
199+
200+
Working with metadata on objects is identical to how it's done with
201+
containers. You use the
202+
:meth:`~openstack.object_store.v1._proxy.Proxy.get_object_metadata` and
203+
:meth:`~openstack.object_store.v1._proxy.Proxy.set_object_metadata` methods.
204+
205+
The metadata attributes to be set can be found on the
206+
:class:`~openstack.object_store.v1.obj.Object` object. ::
207+
208+
>>> secret.delete_after = 300
209+
>>> secret = conn.object_store.set_object_metadata(secret)
210+
211+
We set the :attr:`~openstack.object_store.obj.Object.delete_after`
212+
value to 500 seconds, causing the object to be deleted in 300 seconds,
213+
or five minutes. That attribute corresponds to the ``X-Delete-After``
214+
header value, which you can see is returned when we retreive the updated
215+
metadata. ::
216+
217+
>>> conn.object_store.get_object_metadata(ob)
218+
Object: {'content-length': '11', 'container': u'Secret Container',
219+
'name': u'selfdestruct.txt', 'x-delete-after': 300,
220+
'accept-ranges': 'bytes', 'last-modified': 'Tue, 25 Nov 2014 17:50:45 GMT',
221+
'etag': '5eb63bbbe01eeed093cb22bb8f5acdc3',
222+
'x-timestamp': '1416937844.36805',
223+
'x-trans-id': 'tx5c3fd94adf7c4e1b8f334-005474c17b',
224+
'date': 'Tue, 25 Nov 2014 17:50:51 GMT', 'content-type': 'text/plain'}

‎examples/object_store.py‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Licensed under the Apache License, Version 2.0 (the "License"); you may
2+
# not use this file except in compliance with the License. You may obtain
3+
# a copy of the License at
4+
#
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
#
7+
# Unless required by applicable law or agreed to in writing, software
8+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
9+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
10+
# License for the specific language governing permissions and limitations
11+
# under the License.
12+
13+
from __future__ import print_function
14+
15+
import glob
16+
import os
17+
import sys
18+
19+
from examples import common
20+
from openstack import connection
21+
22+
CONTAINER_HEADER = ("Name{0}| Bytes Used{1}| "
23+
"Num Objects".format(13 * " ", 1 * " "))
24+
CONTAINER_FORMAT = ("{0.name: <16} | {0.bytes: <10} | {0.count}")
25+
OBJECT_HEADER = ("Name{0}| Bytes {1}| "
26+
"Content-Type".format(27 * " ", 2 * " "))
27+
OBJECT_FORMAT = ("{0.name: <30} | {0.bytes: <7} | {0.content_type}")
28+
29+
30+
def list_containers(conn):
31+
print(CONTAINER_HEADER)
32+
print("=" * len(CONTAINER_HEADER))
33+
for container in conn.object_store.containers():
34+
print(CONTAINER_FORMAT.format(container))
35+
36+
37+
def list_objects(conn, container):
38+
print(OBJECT_HEADER)
39+
print("=" * len(OBJECT_HEADER))
40+
for obj in conn.object_store.objects(container.decode("utf8")):
41+
print(OBJECT_FORMAT.format(obj))
42+
43+
44+
def upload_directory(conn, directory, pattern):
45+
"""Upload a directory to object storage.
46+
47+
Given an OpenStack connection, a directory, and a file glob pattern,
48+
upload all files matching the pattern from that directory into a
49+
container named after the directory containing the files.
50+
"""
51+
container_name = os.path.basename(os.path.realpath(directory))
52+
53+
container = conn.object_store.create_container(
54+
container_name.decode("utf8"))
55+
56+
for root, dirs, files in os.walk(directory):
57+
for file in glob.iglob(os.path.join(root, pattern)):
58+
with open(file, "rb") as f:
59+
ob = conn.object_store.create_object(data=f.read(),
60+
obj=file.decode("utf8"),
61+
container=container)
62+
print("Uploaded {0.name}".format(ob))
63+
64+
65+
def main():
66+
# Add on to the common parser with a few options of our own.
67+
parser = common.option_parser()
68+
69+
parser.add_argument("--list-containers", dest="list_containers",
70+
action="store_true")
71+
parser.add_argument("--list-objects", dest="container")
72+
parser.add_argument("--upload-directory", dest="directory")
73+
parser.add_argument("--pattern", dest="pattern")
74+
75+
opts = parser.parse_args()
76+
77+
args = {
78+
'auth_plugin': opts.auth_plugin,
79+
'auth_url': opts.auth_url,
80+
'project_name': opts.project_name,
81+
'domain_name': opts.domain_name,
82+
'project_domain_name': opts.project_domain_name,
83+
'user_domain_name': opts.user_domain_name,
84+
'user_name': opts.user_name,
85+
'password': opts.password,
86+
'verify': opts.verify,
87+
'token': opts.token,
88+
}
89+
conn = connection.Connection(**args)
90+
91+
if opts.list_containers:
92+
return list_containers(conn)
93+
elif opts.container:
94+
return list_objects(conn, opts.container)
95+
elif opts.directory and opts.pattern:
96+
return upload_directory(conn, opts.directory.decode("utf8"),
97+
opts.pattern)
98+
else:
99+
print(parser.print_help())
100+
101+
return -1
102+
103+
if __name__ == "__main__":
104+
sys.exit(main())

0 commit comments

Comments
 (0)