Electronics & Programming

develissimo

Open Source electronics development and programming

  • You are not logged in.
  • Root
  • » PHP
  • » [PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP) [RSS Feed]

#1 Dec. 22, 2007 15:10:27

Christian S.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


Hi,

I was following this thread and came upon Jeff's posting on how closures
could be implemented in PHP.

Since I would find the feature to be EXTREMELY useful, I decided to
actually implement it more or less the way Jeff proposed. So, here's the
patch (against PHP_5_3, I can write one against HEAD if you whish):http://www.christian-seiler.de/temp/closures-php-5-3.patchI started with Wez's patch for adding anonymous functions that aren't
closures. I changed it to make sure no shift/reduce or reduce/reduce
error occur in the grammar. Then I started implementing the actual
closure stuff. It was fun because I learned quite a lot about how PHP
actually works.

I had the following main goals while developing the patch:

1. Don't reinvent the wheel.
2. Don't break anything unless absolutely necessary.
3. Keep it simple.

Jeff proposed a new type of zval that holds additional information about
the function that is to be called. Adding a new type of zval would need
changes throughout the ENTIRE PHP source and probably also throughout
quite a few scripts. But fortunately, PHP already posesses a zval that
supports the storage of arbitrary data while being very lightweight:
Resources. So I simply added a new resource type that stores zend
functions. The $var = function () {}; will now make $var a resource (of
the type "anonymous function".

Anonymous functions are ALWAYS defined at compile time, no matter where
they occur. They are simply named __compiled_lamda_1234 and added to the
global function table. But instead of simply returning the string
'__compiled_lambda_1234', I introduced a new opcode that will create
references to the correct local variables that are referenced inside the
function.

For example, if you have:

$func = function () {
echo "Hello World\n";
};

This will result in an anonymous function called '__compiled_lambda_0'
that is added to the function table at compile time. The opcode for the
assignment to $func will be something like:

1 ZEND_DECLARE_ANON_FUNC ~0 '__compiled_lambda_0'
2 ASSIGN !0, ~0

The ZEND_DECLARE_ANON_FUNC opcode handler does the following:

It creates a new zend_function, copies the contents of the entire
structure of the function table entry corresponding to
'__compiled_lamda_0' into that new structure, increments the refcount,
registeres it as a resource and returns that resource so it can be
assigned to the variable.

Now, have a look at a real closure:

$string = "Hello World!\n";
$func = function () {
lexical $string;
echo $string;
};

This will result in the same opcode as above. But here, three additional
things happen:

1. The compiler sees the keyword 'lexical' and stores the information,
that a variable called 'string' should be used inside the closure.

2. The opcode handler sees that a variable named 'string' is marked as
lexical in the function definition. Therefore it creates a reference to
it in a HashTable of the COPIED zend_function (that will be stored in
the resource).

3. The 'lexical $string;' translates into a FETCH opcode that will work
in exactly the same way as 'static' or 'global' - only fetching it from
the additional HashTable in the zend_function structure.

The resource destructor makes sure that the HashTable containing the
references to the lexical veriables is correctly destroyed upon
destruction of the resource. It does NOT destroy other parts of the
function structure because they will be freed when the function is
removed from the global function table.

With these changes, closures work in PHP.

Some caveats / bugs / todo:

* Calling anonymous functions by name directly is problematic if there
are lexical variables that need to be assigned. I added checks to
make sure this case does not happen.

* In the opcode handler, error handling needs to be added.

* If somebody removes the function from the global function table,
(e.g. with runkit), the new opcode will return NULL instead of
a resource (error handling is missing). Since I do increment
refcount of the zend_function, it SHOULD not cause segfaults or
memory leaks, but I haven't tested it.

* $this is kind of a problem, because all the fetch handlers in PHP
make sure $this is a special kind of variable. For the first version
of the patch I chose not to care about this because what still works
is e.g. the following:

$object = $this;
$func = function () {
lexical $object;
// do something
};

Also, inside the closures, the class context is not preserved, so
accessing private / protected members is not possible.

I'm not sure this actually represents a problem because you can
always use normal local variables to pass values between closure
and calling method and make the calling method change the properties
itself.

* I've had some problems with eval(), have a look at the following
code:

$func = eval ('return function () { echo "Hello World!\n"; };');
$func();

With plain PHP, this seems to work, with the VLD extension loaded
(that shows the Opcodes), it crashes. I don't know if that's a
problem with eval() or just with VLD and I didn't have time to
investigate it further.

* Oh, yes, 'lexical' is now a keyword. Although I really don't think
that TOO many people use that as an identifier, so it probably won't
hurt THAT much.

Except those above points, it really works, even with complex stuff. Let
me show you some examples:

1. Customized array_filter:

function filter_larger ($array, $min = 42) {
$filter = function ($value) {
lexical $min;
return ($value >= $min);
};
return array_filter ($array, $filter);
}

$arr = array (41, 43);
var_dump (filter_larger ($arr)); // 43
var_dump (filter_larger ($arr, 40)); // 41, 43
var_dump (filter_larger ($arr, 44)); // empty

2. Jeff's example:

function getAdder($x) {
return function ($y) {
lexical $x;
return $x + $y;
};
}

$plusFive = getAdder(5);
$plusTen = getAdder(10);

echo $plusFive(4)."\n"; // 9
echo $plusTen(7)."\n"; // 17

3. Nested closures

$outer = function ($value) {
return function () {
lexical $value;
return $value * 2;
};
};

$duplicator = $outer (4);
echo $duplicator ()."\n"; // 8
$duplicator = $outer (8);
echo $duplicator ()."\n"; // 16



It would be great if somebody could review the patch because I'm shure
some parts can still be cleaned up or improved. And it would be even
better if this feature would make it into PHP. ;-)

Regards,
Christian

PS: I'd like to thank Derick Rethans for his GREAT Vulcan Logic
Disassembler - without it, developement would have been a LOT more painful.

PPS: Oh, yeah, if it should be legally necessary, I grant the right to
anybody to use this patch under any OSI certified license you may want
to choose.

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit:http://www.php.net/unsub.php

Offline

#2 Dec. 22, 2007 18:02:54

David Z.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


One question about the names you generate for the function table incombination with opcode caches.Let's assume I have APC installed, and do the following:

foo.php:
$foo = function() {
echo "foo";
}

bar.php:
include('foo.php');

All works fine, and cached versions of both files would be created.


However, if I now change bar.php to this:
$bar = function() {
echo "bar";
}
include('foo.php');wouldn't I get a "cannot redefine function __compiled_lamda_0" fatal,if foo.php was loaded from the opcode cache?While this is a constructed example, it could easily occur withconditional includes with environments that use opcode caches.Or am I completely wrong? :p

- David



Am 22.12.2007 um 16:08 schrieb Christian Seiler:Hi,I was following this thread and came upon Jeff's posting on howclosurescould be implemented in PHP.

Since I would find the feature to be EXTREMELY useful, I decided toactually implement it more or less the way Jeff proposed. So, here'sthepatch (against PHP_5_3, I can write one against HEAD if you whish):http://www.christian-seiler.de/temp/closures-php-5-3.patchI started with Wez's patch for adding anonymous functions that aren't
closures. I changed it to make sure no shift/reduce or reduce/reduce
error occur in the grammar. Then I started implementing the actual
closure stuff. It was fun because I learned quite a lot about how PHP
actually works.

I had the following main goals while developing the patch:

1. Don't reinvent the wheel.
2. Don't break anything unless absolutely necessary.
3. Keep it simple.Jeff proposed a new type of zval that holds additional informationaboutthe function that is to be called. Adding a new type of zval wouldneedchanges throughout the ENTIRE PHP source and probably also throughout
quite a few scripts. But fortunately, PHP already posesses a zval that
supports the storage of arbitrary data while being very lightweight:
Resources. So I simply added a new resource type that stores zendfunctions. The $var = function () {}; will now make $var a resource(ofthe type "anonymous function".Anonymous functions are ALWAYS defined at compile time, no matterwherethey occur. They are simply named __compiled_lamda_1234 and added totheglobal function table. But instead of simply returning the string
'__compiled_lambda_1234', I introduced a new opcode that will createreferences to the correct local variables that are referenced insidethefunction.

For example, if you have:

$func = function () {
echo "Hello World\n";
};

This will result in an anonymous function called '__compiled_lambda_0'that is added to the function table at compile time. The opcode fortheassignment to $func will be something like:

1 ZEND_DECLARE_ANON_FUNC ~0 '__compiled_lambda_0'
2 ASSIGN !0, ~0

The ZEND_DECLARE_ANON_FUNC opcode handler does the following:

It creates a new zend_function, copies the contents of the entire
structure of the function table entry corresponding to
'__compiled_lamda_0' into that new structure, increments the refcount,
registeres it as a resource and returns that resource so it can be
assigned to the variable.

Now, have a look at a real closure:

$string = "Hello World!\n";
$func = function () {
lexical $string;
echo $string;
};This will result in the same opcode as above. But here, threeadditionalthings happen:

1. The compiler sees the keyword 'lexical' and stores the information,
that a variable called 'string' should be used inside the closure.

2. The opcode handler sees that a variable named 'string' is marked aslexical in the function definition. Therefore it creates a referencetoit in a HashTable of the COPIED zend_function (that will be stored in
the resource).3. The 'lexical $string;' translates into a FETCH opcode that willworkin exactly the same way as 'static' or 'global' - only fetching itfromthe additional HashTable in the zend_function structure.

The resource destructor makes sure that the HashTable containing the
references to the lexical veriables is correctly destroyed upon
destruction of the resource. It does NOT destroy other parts of the
function structure because they will be freed when the function is
removed from the global function table.

With these changes, closures work in PHP.

Some caveats / bugs / todo:

* Calling anonymous functions by name directly is problematic if there
are lexical variables that need to be assigned. I added checks to
make sure this case does not happen.

* In the opcode handler, error handling needs to be added.

* If somebody removes the function from the global function table,
(e.g. with runkit), the new opcode will return NULL instead of
a resource (error handling is missing). Since I do increment
refcount of the zend_function, it SHOULD not cause segfaults or
memory leaks, but I haven't tested it.

* $this is kind of a problem, because all the fetch handlers in PHP
make sure $this is a special kind of variable. For the first version
of the patch I chose not to care about this because what still works
is e.g. the following:

$object = $this;
$func = function () {
lexical $object;
// do something
};

Also, inside the closures, the class context is not preserved, so
accessing private / protected members is not possible.

I'm not sure this actually represents a problem because you can
always use normal local variables to pass values between closureand calling method and make the calling method change thepropertiesitself.

* I've had some problems with eval(), have a look at the following
code:

$func = eval ('return function () { echo "Hello World!\n"; };');
$func();

With plain PHP, this seems to work, with the VLD extension loaded
(that shows the Opcodes), it crashes. I don't know if that's a
problem with eval() or just with VLD and I didn't have time to
investigate it further.

* Oh, yes, 'lexical' is now a keyword. Although I really don't think
that TOO many people use that as an identifier, so it probably won't
hurt THAT much.Except those above points, it really works, even with complex stuff.Letme show you some examples:

1. Customized array_filter:

function filter_larger ($array, $min = 42) {
$filter = function ($value) {
lexical $min;
return ($value >= $min);
};
return array_filter ($array, $filter);
}

$arr = array (41, 43);
var_dump (filter_larger ($arr)); // 43
var_dump (filter_larger ($arr, 40)); // 41, 43
var_dump (filter_larger ($arr, 44)); // empty

2. Jeff's example:

function getAdder($x) {
return function ($y) {
lexical $x;
return $x + $y;
};
}

$plusFive = getAdder(5);
$plusTen = getAdder(10);

echo $plusFive(4)."\n"; // 9
echo $plusTen(7)."\n"; // 17

3. Nested closures

$outer = function ($value) {
return function () {
lexical $value;
return $value * 2;
};
};

$duplicator = $outer (4);
echo $duplicator ()."\n"; // 8
$duplicator = $outer (8);
echo $duplicator ()."\n"; // 16



It would be great if somebody could review the patch because I'm shure
some parts can still be cleaned up or improved. And it would be even
better if this feature would make it into PHP. ;-)

Regards,
Christian

PS: I'd like to thank Derick Rethans for his GREAT Vulcan LogicDisassembler - without it, developement would have been a LOT morepainful.PPS: Oh, yeah, if it should be legally necessary, I grant the right to
anybody to use this patch under any OSI certified license you may want
to choose.

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit:http://www.php.net/unsub.php--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit:http://www.php.net/unsub.php

Offline

#3 Dec. 22, 2007 18:48:30

troels k.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


I have another observation about names.
Instead of using an arbitrary name, as the name of the function,
wouldn't it be possible to let the name be derived from the
function-body. Eg., if you took the function-body's tokens and created
a hash from them. This would have two implications: 1) Multiple
definitions of the same function would be optimised into one. And more
importantly 2) , it would be possible to serialize/unserialize a
closure. Of course, this won't work if an anonymous function is a
resource, since resources can't be serialized. This would work for
Wez' original patch though.

--
troels

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit:http://www.php.net/unsub.php

Offline

#4 Dec. 22, 2007 19:44:25

Martin A.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


2007/12/22, Christian Seiler <>:
>
>
> PPS: Oh, yeah, if it should be legally necessary, I grant the right to
> anybody to use this patch under any OSI certified license you may want
> to choose.
>
>
That's very kind of you but, if I was explained right, you don't have
copyright on a patch. If I understood the legalese correctly, you retain
authorship on the code provided, but this kind of derived work, which has no
purpose outside the original, cannot be claimed through copyright. You
specially did it for the php codebase, and has not a meaning outside it.

Consider what happens to the code you make to maintain an in-house
application of a company. You cannot claim copyright on that codebase, it
remains to the company. But, if you were to develop an app from scratch for
the same company, the company has the right to use it in-house, but if they
want to distribute it, they have to require your consent, because you're one
of the copyright holders.

Also consider what happens to reviewers that check and correct works from
other people, such as syntax and grammar checking in literature, or code
reviewing in software. They can claim authorship on their corrections, but
they cannot claim copyright.

Anyway, it's too complicate. The bottom line is that copyright cannot be
claimed on patches. Which makes complete sense, imagine what would happen to
open-source if it wasn't this way... the licensing documentation would
occupy more than the code itself...

Offline

#5 Dec. 23, 2007 20:53:12

Martin A.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


2007/12/22, Christian Seiler <>:
>
> Hi,
>
> I was following this thread and came upon Jeff's posting on how closures
> could be implemented in PHP.
>
> Since I would find the feature to be EXTREMELY useful, I decided to
> actually implement it more or less the way Jeff proposed. So, here's the
> patch (against PHP_5_3, I can write one against HEAD if you whish):
>
>http://www.christian-seiler.de/temp/closures-php-5-3.patch>

Hi,

I finally got some time to check out your patch (the first and the second
version). It looks cool, works as expected in most of the things I've tried.

Nevertheless, I don't need this feature, it doesn't provide any advantage
over the functor objects I've been using since long. Closures and anonymous
functions are harder to maintain and debug, and they are not as elegant as
their counterparts in functional languages (lambda functions). On the other
side, functor objects are more maintainable, scalable, elegant, and
versatile (and php already supports them... it could be improved though).

Bottom line, excellent work. If a considerable part of the php user base
need this, excellent, they have an implementation which seems viable. But,
sorry, I couldn't care less about closures or anonymous functions. Anyway,
my opinion doesn't matter, I'm just one user.

Best Regards,

Martin Alterisio

Offline

#6 March 23, 2008 15:46:04

Marcus B.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

[PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP)


Hello Christian,

I have put your proposal as a link to a PHP GSoC 2008 idea here:http://wiki.php.net/gsoc/2008Feel invited to add to this idea in whatever way you want :-)

marcus

Saturday, December 22, 2007, 4:08:04 PM, you wrote:

> Hi,

> I was following this thread and came upon Jeff's posting on how closures
> could be implemented in PHP.

> Since I would find the feature to be EXTREMELY useful, I decided to
> actually implement it more or less the way Jeff proposed. So, here's the
> patch (against PHP_5_3, I can write one against HEAD if you whish):

>http://www.christian-seiler.de/temp/closures-php-5-3.patch> I started with Wez's patch for adding anonymous functions that aren't
> closures. I changed it to make sure no shift/reduce or reduce/reduce
> error occur in the grammar. Then I started implementing the actual
> closure stuff. It was fun because I learned quite a lot about how PHP
> actually works.

> I had the following main goals while developing the patch:

> 1. Don't reinvent the wheel.
> 2. Don't break anything unless absolutely necessary.
> 3. Keep it simple.

> Jeff proposed a new type of zval that holds additional information about
> the function that is to be called. Adding a new type of zval would need
> changes throughout the ENTIRE PHP source and probably also throughout
> quite a few scripts. But fortunately, PHP already posesses a zval that
> supports the storage of arbitrary data while being very lightweight:
> Resources. So I simply added a new resource type that stores zend
> functions. The $var = function () {}; will now make $var a resource (of
> the type "anonymous function".

> Anonymous functions are ALWAYS defined at compile time, no matter where
> they occur. They are simply named __compiled_lamda_1234 and added to the
> global function table. But instead of simply returning the string
> '__compiled_lambda_1234', I introduced a new opcode that will create
> references to the correct local variables that are referenced inside the
> function.

> For example, if you have:

> $func = function () {
> echo "Hello World\n";
> };

> This will result in an anonymous function called '__compiled_lambda_0'
> that is added to the function table at compile time. The opcode for the
> assignment to $func will be something like:

> 1 ZEND_DECLARE_ANON_FUNC ~0 '__compiled_lambda_0'
> 2 ASSIGN !0, ~0

> The ZEND_DECLARE_ANON_FUNC opcode handler does the following:

> It creates a new zend_function, copies the contents of the entire
> structure of the function table entry corresponding to
> '__compiled_lamda_0' into that new structure, increments the refcount,
> registeres it as a resource and returns that resource so it can be
> assigned to the variable.

> Now, have a look at a real closure:

> $string = "Hello World!\n";
> $func = function () {
> lexical $string;
> echo $string;
> };

> This will result in the same opcode as above. But here, three additional
> things happen:

> 1. The compiler sees the keyword 'lexical' and stores the information,
> that a variable called 'string' should be used inside the closure.

> 2. The opcode handler sees that a variable named 'string' is marked as
> lexical in the function definition. Therefore it creates a reference to
> it in a HashTable of the COPIED zend_function (that will be stored in
> the resource).

> 3. The 'lexical $string;' translates into a FETCH opcode that will work
> in exactly the same way as 'static' or 'global' - only fetching it from
> the additional HashTable in the zend_function structure.

> The resource destructor makes sure that the HashTable containing the
> references to the lexical veriables is correctly destroyed upon
> destruction of the resource. It does NOT destroy other parts of the
> function structure because they will be freed when the function is
> removed from the global function table.

> With these changes, closures work in PHP.

> Some caveats / bugs / todo:

> * Calling anonymous functions by name directly is problematic if there
> are lexical variables that need to be assigned. I added checks to
> make sure this case does not happen.

> * In the opcode handler, error handling needs to be added.

> * If somebody removes the function from the global function table,
> (e.g. with runkit), the new opcode will return NULL instead of
> a resource (error handling is missing). Since I do increment
> refcount of the zend_function, it SHOULD not cause segfaults or
> memory leaks, but I haven't tested it.

> * $this is kind of a problem, because all the fetch handlers in PHP
> make sure $this is a special kind of variable. For the first version
> of the patch I chose not to care about this because what still works
> is e.g. the following:

> $object = $this;
> $func = function () {
> lexical $object;
> // do something
> };

> Also, inside the closures, the class context is not preserved, so
> accessing private / protected members is not possible.

> I'm not sure this actually represents a problem because you can
> always use normal local variables to pass values between closure
> and calling method and make the calling method change the properties
> itself.

> * I've had some problems with eval(), have a look at the following
> code:

> $func = eval ('return function () { echo "Hello World!\n"; };');
> $func();

> With plain PHP, this seems to work, with the VLD extension loaded
> (that shows the Opcodes), it crashes. I don't know if that's a
> problem with eval() or just with VLD and I didn't have time to
> investigate it further.

> * Oh, yes, 'lexical' is now a keyword. Although I really don't think
> that TOO many people use that as an identifier, so it probably won't
> hurt THAT much.

> Except those above points, it really works, even with complex stuff. Let
> me show you some examples:

> 1. Customized array_filter:

> function filter_larger ($array, $min = 42) {
> $filter = function ($value) {
> lexical $min;
> return ($value >= $min);
> };
> return array_filter ($array, $filter);
> }

> $arr = array (41, 43);
> var_dump (filter_larger ($arr)); // 43
> var_dump (filter_larger ($arr, 40)); // 41, 43
> var_dump (filter_larger ($arr, 44)); // empty

> 2. Jeff's example:

> function getAdder($x) {
> return function ($y) {
> lexical $x;
> return $x + $y;
> };
> }

> $plusFive = getAdder(5);
> $plusTen = getAdder(10);

> echo $plusFive(4)."\n"; // 9
> echo $plusTen(7)."\n"; // 17

> 3. Nested closures

> $outer = function ($value) {
> return function () {
> lexical $value;
> return $value * 2;
> };
> };

> $duplicator = $outer (4);
> echo $duplicator ()."\n"; // 8
> $duplicator = $outer (8);
> echo $duplicator ()."\n"; // 16

>

> It would be great if somebody could review the patch because I'm shure
> some parts can still be cleaned up or improved. And it would be even
> better if this feature would make it into PHP. ;-)

> Regards,
> Christian

> PS: I'd like to thank Derick Rethans for his GREAT Vulcan Logic
> Disassembler - without it, developement would have been a LOT more painful.

> PPS: Oh, yeah, if it should be legally necessary, I grant the right to
> anybody to use this patch under any OSI certified license you may want
> to choose.




Best regards,
Marcus


--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit:http://www.php.net/unsub.php

Offline

  • Root
  • » PHP
  • » [PHP-DEV] PATCH: Implementing closures in PHP (was: anonymous functions in PHP) [RSS Feed]

Board footer

Moderator control

Enjoy the 11th of December
PoweredBy

The Forums are managed by develissimo stuff members, if you find any issues or misplaced content please help us to fix it. Thank you! Tell us via Contact Options
Leave a Message
Welcome to Develissimo Live Support