1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
|
Cppy smart pointer
==================
CPython relies on reference counting to manage object lifetime. A large pitfall
when writing C-extension is to properly handle increfing and decrefing the
reference count. Cppy aims at simplifying this process by providing a smart
pointer class. Before diving into the details of it helps lets start a CPython
reference counting crash-course.
CPython reference counting crash course
---------------------------------------
Each object allocated by Python has a reference count, indicating how many
times this object is 'used'. When the reference count of an object goes to
zero, it is de-allocated. Outside of C extension, one does not need to manage
the reference count manually.
When a function part of Python C-API returns a Python object, it returns a
pointer to it. At the time at which the function returns, the referenced
object is live and its reference count is above zero. Depending of the
function, you do not have the same responsibility with respect to that object
reference count:
- Owned references:
Most functions return a ``new`` reference which means that you are responsible
for decrefing the object reference count when you are done with it (basically
the function increfed the object reference count before returning).
In this situation you own a reference.
- Borrowed reference:
Some functions (``PyList_GetItem``, ``PyTuple_GetItem``, ``PyDict_GetItem``, ...)
do not incref the object count before returning. In that case, you have only
a borrowed reference, you are not responsible for decrefing the object
reference count.
Borrowed references allow to avoid the cost of increfing/decrefing which is
nice. However since you do not own the reference, if the object referenced is
removed from its owner (list, tuple for the above two mentioned functions) it
may just disappear and your reference becomes invalid. This can cause issues.
If the object should outlive the container, or the time it will spend in the
container you have to incref it manually. Lets now discuss the convention when
calling a function.
When calling a function, the caller is expected to own a reference to each of
the arguments passed to the callee. The callee does not own the references, it
only borrows them. As a consequence, it should not decref the reference and if
it needs to store the object, in for example a C structure, it should incref
it. Note that this does not apply in general to Python container since those
are manipulated using functions that take care of it. There are however some
exceptions that steals a reference, meaning that you are not the owner of the
reference after the call. ``PyList_SetItem``, for example, steal references.
An easy way to get reference count wrong is forgetting to decref some
intermediate object before leaving a function. This is particularly true if the
function has some early exit point because an exception should be raised. A
good practice is to have a single exit point, however it is not always
possible/practical and even like this it is possible to miss references, this
is typically where cppy can help.
This is a very brief introduction to reference counting. You can read a bit
more in the official `Python documentation`_ and in the `Python API`_
documentation.
.. _Python documentation: https://docs.python.org/3/c-api/intro.html#objects-types-and-reference-counts
.. _Python API: https://docs.python.org/3/c-api/refcounting.html
Cppy smart pointer class
------------------------
Cppy smart pointer (``cppy::ptr``) can be initialized with a pointer to a Python
object as follows:
.. code:: c++
cppy::ptr obj_ptr( PyUnicode_FromString("test") )
When created, the class assume that you own the reference, if it is not the
case you should incref it first:
.. code:: c++
PyObject* function( PyObject* obj )
{
cppy::ptr obj_ptr( cppy::incref( obj ) );
cppy::ptr obj_ptr2( obj, true );
}
.. note::
Cppy provides convenient inline function for common reference manipulation:
- ``cppy::incref``, ``cppy::xincref``, ``cppy::decref``, ``cppy::xdecref`` use the
the similarly named Python macros and return the input value.
- ``cppy::clear``, ``cppy::replace`` are similar but return void.
You can also initialize a ``cppy::ptr`` from another ``cppy::ptr`` in which case
the reference count will always be incremented.
The main advantage provided by ``cppy::ptr`` is that it implements a destructor
that will be invoked automatically by the c++ runtime when the ``cppy::ptr``
goes out of scope. The destructor will decref the reference for you. As a
consequence you can be sure that your reference you always be decremented when
you leave the function.
Sometimes, however, that is not what you want, because you want to return the
reference the ``cppy::ptr`` manage. You can request the ``cppy::ptr`` to give back
the reference using its ``release`` method. Lets illustrate on a tiny example:
.. code:: c++
PyObject* function( PyObject* obj )
{
cppy::ptr repr_ptr( PyObject_Repr( obj ) );
return repr_ptr.release();
}
Function which are part of Python C-API are not aware of of ``cppy::ptr`` and
when calling them you need to provide the original ``PyObject*``. To access, you
simply need to call the ``get`` method of the ``cppy::ptr`` object.
.. code:: c++
PyObject* function( PyObject* obj )
{
cppy::ptr l_ptr( PyList_New() );
if( PyList_Append( l_ptr.get(), obj ) != 0 )
return 0;
return l_ptr.release();
}
Here we see that because we use ``cppy::ptr`` to manage the list, we do not have
to worry about decrefing the reference if an exception occurs, the runtime
will do it for us. If no exception occurs, we stop managing the reference and
we are good.
Using cppy does not eliminate all the pitfalls of writing C-extensions. For
example if you release too early (for example when passing the object to a
function that may fail), you can still leak references. However it does
alleviate some of the complexity.
``cppy::ptr`` methods
---------------------
All methods that takes a ``PyObject*`` can also accept a ``cppy::ptr``.
Most names should be self-explanatory, and apart from the is\_ methods most of
them rely on the PyObject\_ functions similarly named:
.. code:: c++
bool is_none() const
bool is_true() const
bool is_false() const
bool is_bool() const
bool is_int() const
bool is_float() const
bool is_list() const
bool is_dict() const
bool is_set() const
bool is_bytes() const
bool is_str() const
bool is_unicode() const
bool is_callable() const
bool is_iter() const
bool is_type( PyTypeObject* cls ) const
int is_truthy() const
int is_instance( PyObject* cls ) const
int is_subclass( PyObject* cls ) const
PyObject* iter() const
PyObject* next() const
PyObject* repr() const
PyObject* str() const
PyObject* bytes() const
PyObject* unicode() const
Py_ssize_t length() const
PyTypeObject* type() const
int richcmp( PyObject* other, int opid ) const
long hash() const
bool hasattr( PyObject* attr ) const
bool hasattr( const char* attr ) const
bool hasattr( const std::string& attr ) const
PyObject* getattr( PyObject* attr ) const
PyObject* getattr( const char* attr ) const
PyObject* getattr( const std::string& attr ) const
bool setattr( PyObject* attr, PyObject* value ) const
bool setattr( const char* attr, PyObject* value ) const
bool setattr( const std::string& attr, PyObject* value ) const
bool delattr( PyObject* attr ) const
bool delattr( const char* attr ) const
bool delattr( const std::string& attr ) const
PyObject* getitem( PyObject* key ) const
bool setitem( PyObject* key, PyObject* value ) const
bool delitem( PyObject* key )
PyObject* call( PyObject* args, PyObject* kwargs = 0 ) const
|