Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
12cda74
[LIB-837][WIP] working on custom sockets in LIB
opalczynski Aug 10, 2016
460006f
[LIB-837] Add missing docs about custom_sockets; move utils to anothe…
opalczynski Aug 11, 2016
5eca906
[LIB-837] add missing fields in doc string;
opalczynski Aug 11, 2016
c0f75cb
[LIB-837] correct documentation; add possibility to read all endpoint…
opalczynski Aug 12, 2016
e108b5a
[LIB-837] Correct after tests;
opalczynski Aug 12, 2016
5eab464
[LIB-837] another portion of corrects;
opalczynski Aug 12, 2016
1afcd9f
[LIB-837] correct tests again;
opalczynski Aug 12, 2016
48eb08e
[LIB-837] change tests instead of doing a magic in ModelField;
opalczynski Aug 12, 2016
4e2ee33
Language updates
MariuszWisniewski Aug 12, 2016
fe52e7a
[ci skip] readme
MariuszWisniewski Aug 12, 2016
7bc974a
[ci skip] Endpoints readme update
MariuszWisniewski Aug 15, 2016
bc71ad1
[LIB-837] revert ModelField on script in script endpoint model;
opalczynski Aug 16, 2016
327a3fc
[LIB-837] correct data passing on run methods;
opalczynski Aug 16, 2016
d50344b
[LIB-837] correct custom socket behaviour;
opalczynski Aug 16, 2016
c2ffd3a
Merge branch 'LIB-837' of github.com:Syncano/syncano-python into LIB-837
opalczynski Aug 16, 2016
42e4373
[LIB-837] corrects after qa;
opalczynski Aug 16, 2016
04c5ef6
[LIB-837] corrects after qa;
opalczynski Aug 16, 2016
9f5adba
[LIB-837] correct docs examples, polish the tests;
opalczynski Aug 16, 2016
69f3a11
[LIB-837] remove default custom socket in setupclass;
opalczynski Aug 16, 2016
3d1f4dd
[LIB-837] remove default custom socket in setupclass;
opalczynski Aug 16, 2016
d982fd0
[LIB-837] final test correction;
opalczynski Aug 16, 2016
8662e42
Update language
devintyler Aug 16, 2016
0864163
[LIB-837] add possibility to directly point instance name when gettin…
opalczynski Aug 18, 2016
9246b98
[LIB-837] change interface: publish to install in custom socket context;
opalczynski Aug 19, 2016
31c51b9
Merge branch 'LIB-837' of github.com:Syncano/syncano-python into LIB-837
opalczynski Aug 19, 2016
14f6ff7
[LIB-837] add status badges to README
opalczynski Aug 19, 2016
2372986
[LIB-837] small changes after CORE change;
opalczynski Aug 22, 2016
515e50b
[LIB-837] correct SocketEndpoint behaviour (run mainly);
opalczynski Aug 22, 2016
16795dd
[LIB-837] correct run in CustomSocket;
opalczynski Aug 22, 2016
b8c252f
[LIB-837] add possiblity to install from url;
opalczynski Aug 23, 2016
3cf2465
[LIB-837] update documentation for installing socket from url;
opalczynski Aug 23, 2016
ef39be6
[LIB-837] fixes after qa;
opalczynski Aug 24, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[LIB-837] Add missing docs about custom_sockets; move utils to anothe…
…r file, add possibility to recheck and update custom socket; add possibility to remove endpoints and dependencies;
  • Loading branch information
opalczynski committed Aug 11, 2016
commit 460006f22610a843b63ffcc982da804ac3c427bc
214 changes: 214 additions & 0 deletions docs/source/custom_sockets.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
.. _custom-sockets:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lifcio We need help from native here ;)


=========================
Custom Sockets in Syncano
=========================

``Syncano`` provides possibility of creating the custom sockets. It means that there's a possibility
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

creating the custom sockets

to define a very specific endpoints in syncano application and use them as normal api calls.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use API not api to be more correct here (with it being an acronym and all). Same for a few other uses below.

Currently custom sockets allow only one dependency - script. This mean that on the backend side
each time the api is called - the script is executed and result from this script is returned as a result of the
api call.

Creating a custom socket
------------------------

There are two methods of creating the custom socket. First: use the helpers objects defined in Python Libray.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's still the one and only method. Library simply provides some helpers for it. Indicating that it is separate somehow and is overall a different method - can lead to some confusion. Even I'm confused after reading this ;) especially when it says about describing raw format below and then goes on to using python helpers. I think we can just skip this paragraph and move on to an example about how to actually create it

Second: use the raw format - this is described below.

To create a custom socket follow the steps::
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow the these steps
Well, these aren't exactly steps. But more like an example. But if we add some comments below it will be more "steps-like" ;)


import syncano
from syncano.models import CustomSocket, Endpoint, ScriptCall, ScriptDependency, RuntimeChoices
from syncano.connection import Connection

custom_socket = CustomSocket(name='my_custom_socket') # this will create an object in place (do api call)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add comments about actual steps numbered.
Above here:
# 1. Create Socket object


# define endpoints
Copy link
Copy Markdown
Member

@23doors 23doors Aug 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# 2. Define endpoints
and so son

my_endpoint = Endpoint(name='my_endpoint') # again - no api call here
my_endpoint.add_call(ScriptCall(name='custom_script'), methods=['GET'])
my_endpoint.add_call(ScriptCall(name='another_custom_script'), methods=['POST'])

# explanation for the above lines:
# The endpoint will be seen under `my_endpoint` name:
# On this syncano api endpoint the above endpoint will be called (after custom socket creation)
# <host>://<api_version>/instances/<instance_name>/endpoints/sockets/my_endpoint/
# On this syncano api endpoint the details of the defined endpoint will be returned
# <host>://<api_version>/instances/<instance_name>/sockets/my_custom_socket/endpoints/my_endpoint/
# For the above endpoint - the two calls are defined, one uses GET method - the custom_script will be executed
# there, second uses the POST method and then the another_custom_script will be called;
# Currently only script are available for calls;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand above comments. Could someone rewrite them in other words in here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every script consists of dependencies (which are simply checked and created when needed) and endpoints.
Here:

    my_endpoint = Endpoint(name='my_endpoint')
    my_endpoint.add_call(ScriptCall(name='custom_script'), methods=['GET'])
    my_endpoint.add_call(ScriptCall(name='another_custom_script'), methods=['POST'])

we create one endpoint called my_endpoint and we define that: when someone does GET my_endpoint -> call script with endpoint=custom_script. When someone does POST my_endpoint, call script with endpoint=another_custom_script. So we can define separate logic/bindings based on a HTTP method used.

Now endpoints live in two places in API.
/instances/<instance_name>/sockets/my_custom_socket/endpoints/
Here we will get them as REST objects and we will only get endpoints created by my_custom_socket.
So our endpoint will have URI: /instances/<instance_name>/sockets/my_custom_socket/endpoints/my_endpoint/
but this will just be a read_only REST object (with a link to URI that can run it).

Second place defines API for custom sockets in general.
/instances/<instance_name>/endpoints/sockets/
Here you will get all endpoints created by all sockets. This will also be a list of read only REST objects, but if you do an action on a specific endpoint resource URI, e.g.:
GET /instances/<instance_name>/endpoints/sockets/my_endpoint/
it will instead be used as a call to that endpoint and as a result - call Script custom_script. While POST will call Script another_custom_script (as these are the only 2 actions that we defined).

It's a little verbose but hopefully I explained it enough?
You could also check: https://github.com/Syncano/syncano-platform/pull/1240 (last 2 sections, endpoints and socket endpoints access)


# After the creation of the endpoint, add them to custom_socket:
custom_socket.add_endpoint(my_endpoint)

# define dependency now;
# using a new script - defining new source code;
custom_socket.add_dependency(
ScriptDependency(
Script(
label='custom_script',
runtime_name=RuntimeChoices.PYTHON_V5_0,
source='print("custom_script")'
)
)
)
# using a existing script:
another_custom_script = Script.please.get(id=2)
custom_socket.add_dependency(
ScriptDependency(
another_custom_script
)
)

# now it is time to publish custom_socket;
custom_socket.publish() # this will do an api call and will create script;

Some time is needed to setup the environment for this custom socket.
There is possibility to check the custom socket status::

print(custom_socket.status)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will always be "checking" after creation/update so maybe here first .reload() then print

# and
print(custom_socket.status_info)

# to reload object (read it again from syncano api) use:
custom_socket.reload()



Updating the custom socket
--------------------------

To update custom socket, use::

custom_socket = CustomSocket.please.get(name='my_custom_socket')

custom_socket.remove_endpoint(endpoint_name='my_endpoint')
custom_socket.remove_dependency(dependency_name='custom_script')

# or add new:

custom_socket.add_endpoint(new_endpoint) # see above code for endpoint examples;
custom_socket.add_dependency(new_dependency) # see above code for dependency examples;

custom_socket.update()


Running the custom socket
-------------------------

To run custom socket use::

# this will run the my_endpoint - and call the custom_script (method is GET);
result = custom_socket.run(method='GET', endpoint_name='my_endpoint')


Read all endpoints
------------------

To get the all defined endpoints in custom socket run::

endpoints = custom_socket.get_endpoints()

for endpoint in endpoints:
print(endpoint.name)
print(endpoint.calls)

To run particular endpoint::
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To run a particular endpoint


endpoint.run(method='GET')
# or:
endpoint.run(method='POST', data={'name': 'test_name'})

The data will be passed to the api call in the request body.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about getting list of all socket endpoints (not bound to specific socket)?
basically GET /v1.1/instances/xxx/endpoints/sockets/
That could be more useful as it doesn't require to get specific socket.

It would also be useful if user could run some custom endpoint without getting anything at all.
E.g.

result = SocketEndpoint('some_endpoint').run()  # run GET on /endpoints/sockets/some_endpoint 
result = SocketEndpoint('some_endpoint').run(method='POST', data={'a': 'b'})  # run POST on /endpoints/sockets/some_endpoint 

This may end up as how it will be used the most in apps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++


Custom sockets endpoints
------------------------

Each custom socket is created from at least one endpoint. The endpoint is characterized by name and
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not from.
Each custom socket requires to define at least one endpoint.

Also, instead of characterized in these whole docs, defined may be a little more commonly used and fitting

defined calls. Calls is characterized by name and methods. The name is a identification for dependency, eg.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and a list of calls. Each call is defined by a name and a list of methods.

if it's equal to 'my_script' - the Script with label 'my_script' will be used (if exist and the source match),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not exactly.
the Script Endpoint with name 'my_script' will be used (if it exists and Script source and runtime matches),

or new one will be created.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or a new one

There's a special wildcard method: `methods=['*']` - this mean that any request with
any method will be executed in this endpoint.

To add endpoint to the custom_socket use::
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To add an endpoint


my_endpoint = Endpoint(name='my_endpoint') # again - no api call here
my_endpoint.add_call(ScriptCall(name='custom_script'), methods=['GET'])
my_endpoint.add_call(ScriptCall(name='another_custom_script'), methods=['POST'])

custom_socket.add_endpoint(my_endpoint)

Custom socket dependency
------------------------

Each custom socket has dependency - this is a meta information for endpoint: which resource
should be used to return the api call results. The dependencies are bind to the endpoints call objects.
Currently supported dependency in only script.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ordering is strange ;)
Currently the only supported dependency is script.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yoda style ;)


**Using new script**

::

custom_socket.add_dependency(
ScriptDependency(
Script(
label='custom_script',
runtime_name=RuntimeChoices.PYTHON_V5_0,
source='print("custom_script")'
)
)
)


**Using defined script**

::

another_custom_script = Script.please.get(id=2)
custom_socket.add_dependency(
ScriptDependency(
another_custom_script
)
)


Custom socket recheck
---------------------

The creation of the socket can fail - this happen, eg. when endpoint name is already taken by another
custom socket. To check the statuses use::

print(custom_socket.status)
print(custom_socket.status_info)

There is a possibility to re-check socket - this mean that if conditions are met - the socket will be
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, dependencies can be mistakenly deleted. So re-check is useful here to well, "reinstall" everything that is missing. The socket itself is not actually created again as it's already there. It just checks endpoints and dependencies again.

`created` again and available to use - if not the error will be returned in status field.

Custom socket - raw format
--------------------------

There is a possibility to create a custom socket from the raw JSON format::
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like:
If you prefer raw JSON format for creating sockets, you can resort to use it in python library as well::
to indicate that while it's another possibility, it comes down to some preferences that aren't exactly the main purpose here


CustomSocket.please.create(
name='my_custom_socket_3',
endpoints={
"my_endpoint_3": {
"calls":
[
{"type": "script", "name": "my_script_3", "methods": ["POST"]}
]
}
},
dependencies=[
{
"type": "script",
"runtime_name": "python_library_v5.0",
"name": "my_script_3",
"source": "print(3)"
}
]
)

The disadvantage of this method is that - the JSON internal structure must be known by developer.
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Contents:

getting_started
interacting
custom_sockets
refs/syncano


Expand Down
7 changes: 7 additions & 0 deletions docs/source/refs/syncano.models.custom_sockets.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
syncano.models.custom_sockets
=============================

.. automodule:: syncano.models.custom_sockets
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions docs/source/refs/syncano.models.custom_sockets_utils.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
syncano.models.custom_sockets_utils
===================================

.. automodule:: syncano.models.custom_sockets_utils
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions docs/source/refs/syncano.models.geo.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
syncano.models.geo
==================

.. automodule:: syncano.models.geo
:members:
:undoc-members:
:show-inheritance:
7 changes: 7 additions & 0 deletions docs/source/refs/syncano.models.hosting.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
syncano.models.hosting
======================

.. automodule:: syncano.models.hosting
:members:
:undoc-members:
:show-inheritance:
3 changes: 2 additions & 1 deletion syncano/models/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@
from .backups import * # NOQA
from .hosting import * # NOQA
from .data_views import DataEndpoint as EndpointData # NOQA
from .custom_sockets import * # NOQA
from .custom_sockets import * # NOQA
from .custom_sockets_utils import Endpoint, ScriptCall, ScriptDependency # NOQA
Loading