AGPL enforced: The Israeli ICT authority releases code

Data.gov.il was created in 2011 after the Israeli social justice protests as part of the the public participation initiative and started to offer data held by the government. Back then the website was based on Drupal. In 2016 it was changed to CKAN, a designated system for releasing data. This system is licensed under the AGPLv3 requiring source code availability for anyone who can access the the system over a network, de facto for every user.

Since the change to CKAN, open source people asked the state to release the code according to the license but didn’t get a clear answer. All this time when it’s clear it’s violation.  This led Gai Zomer to file a formal complaint in March 2017 with the Israeli State Comptroller. Absurdly, that same month the ICT authority mentioned a policy to release source code it owns, while failing to release code it has taken from others and adapted.

With the end of the summer break and Jew holidays, and after I wasn’t able to get the source, I decided to switch to legal channels, and with the help of Jonathan Klinger and my company, Kaplan Open Source Consulting, we notified they should provide the source code or we’ll address the court.

Well, it worked. In 3 days time the CKAN extensions where available on the website, but in a problematic way, so users weren’t able to download easily. This is why we decided not to publish this code release and let them fix it first. In addition we made it clear all the source code should be available, not only the extensions. Further more, if they already release it’s recommended to use git format instead of just “dumping” a tarball. So we told them if they aren’t going to make a git repository we’ll do that ourselves, but in any case, would prefer them to do that .

While this issue is still pending, the ICT authority had a conference called “the citizen 360” about e-gov and open government in which they reaffirmed their open source plans.

A slide about open source from the Israeli ICT authority presentation

Now, a month later, after our second letter to them, the about page in data.gov.il was updated with links to the ICT authority GitHub account which has the sources for the website and the extensions. A big improvement, and an important mark point as the commit to the repository was done by an official (gov.il) email address.

Beyond congratulating the Israeli ICT authority for their steps forward and the satisfaction of our insisting on them became fruitful, we would like to see the repository get updated on a regular basis, the code being given back to the various CKAN extensions (e.g. Hebrew translation). In general, we hope they would to get inspired by how the how data.gov.uk is doing technical transparency. If we allow ourselves to dream, we would like to see Israel becoming a dominate member in the CKAN community and among the other governments who use it.

We’re happy to be the catalyst for open source in the Israeli government, and we promise to keep insisted where needed. We know that due to other requests and notifications more organizations are on their way to release code.

(This post is a translation from Hebrew of a post in Kaplan Open Source Consulting at https://kaplanopensource.co.il/2017/11/20/data-gov-il-code-release/)

PHP 7.2 is coming… mcrypt extension isn’t

Early September, it’s about 3 months before PHP 7.2 is expected to be release (schedule here). One of the changes is the removal of the mcrypt extension after it was deprecated in PHP 7.1. The main problem with mcrypt extension is that it is based on libmcrypt that was abandoned by it’s upstream since 2007. That’s 10 years of keeping a library alive, moving the burden to distribution’s security teams. But this isn’t new, Remi already wrote about this two years ago: “About libmcrypt and php-mcrypt“.

But with removal of the extension from the PHP code base (about F**King time), it would force the recommendation was done “nicely” till now. And forcing people means some noise, although an alternative is PHP’s owns openssl extension. But as many migrations that require code change – it’s going slow.

The goal of this post is to reach to the PHP eco system and map the components (mostly frameworks and applications) to still require/recommend mcyrpt and to pressure them to fix it before PHP 72 is released. I’ll appreciate the readers’ help with this mapping in the comments.

For example, Laravel‘s release notes for 5.1:

In previous versions of Laravel, encryption was handled by the mcrypt PHP extension. However, beginning in Laravel 5.1, encryption is handled by the openssl extension, which is more actively maintained.

Or, on the other hand Joomla 3 requirements still mentions mcrypt.

mcrypt safe:

• Drupal 7 and up, see https://www.drupal.org/docs/7/system-requirements/php
• Lavavel 5.1 and up, see https://laravel.com/docs/5.1/releases

mcrypt dependant:

• Joomla, see https://downloads.joomla.org/technical-requirements
• Magento, see http://devdocs.magento.com/guides/v2.2/install-gde/system-requirements-tech.html
  (Checking 2.2 RC release, as it just added support for PHP 7.1)

For those who really need mcrypt, it is part of PECL, PHP’s extensions repository. You’re welcome to compile it on your own risk.

PDO_IBM: tracking changes publicly

As part of my work at Zend (now a RogueWave company), I maintain the various patch sets. One of those is the changes for PDO_IBM extension for PHP.

After some patch exchange I decided it’s would be easier to manage the whole process over a public git repository, and maybe gain some more review / feedback along the way. Info at https://github.com/kaplanlior/pecl-database-pdo_ibm/commits/zend-patches

Another aspect of this, is having IBMi specific patches from YIPS (young i professionals) at http://www.youngiprofessionals.com/wiki/index.php/XMLService/PHP, which itself are patches on top of vanilla releases. Info at https://github.com/kaplanlior/pecl-database-pdo_ibm/commits/zend-patches-for-yips

So keeping track over these changes as well is easier while using git’s ability to rebase efficiently, so when a new release is done, I can adapt my patches quite easily. Make sure the changes can be back and forward ported between vanilla and IBMi versions of the extension.

First uses of the PHP 5.4 security backports

I recently checked the Debian PHP 5.4 changelog and found out this message (5.4.45-0+deb7u3 and 5.4.45-0+deb7u4):

* most patches taken from https://github.com/kaplanlior/php-src
Thanks a lot to Lior Kaplan for providing them.

I was very pleased to see my work being used, and I hope this would save other time while providing PHP long term support.

Also, while others do similar work (e.g. Remi from RedHat), it seems I’m the only when that make this over GIT and with full references (e.g. commit info, CVE info and bug number).

Comments and suggestions are always welcome… either mail or even better – a pull request.

Anonymous CVE requests

A year ago I’ve blogged about people requesting CVE without letting upstream know. On the other hand, per requests from Debian, I’m working on improving PHP upstream CVE request process. For the last few release this means I ask the security list members which issues they think should have CVE and ask for them in parallel to the release being made (usually in the space between the release being tagged publicly and is actually announced).

In the last week, I’ve encountered a case where a few CVE were assigned to old PHP issues without any public notice. The fixes for these issues have been published a year ago (August 2015). And I find out about these assignment through warning published by the distributions (mostly Debian, which I’m close to).

Sometimes things fall between the chairs, and it’s perfectly OK to ask for CVE to make sure security issues do get attention even if time has passed. But after the issues (and fixes) are public, I don’t see a reason to do so without making the request itself public as well. And even if the request wasn’t public, at least notify upstream so this info can be added to the right places. Most of these bug were found out when I started to add sequential number into the CVE search after getting an a notice from Debian for two of the issues.

• CVE-2015-8873 for PHP #69793
• CVE-2015-8874 for PHP #66387
• CVE-2015-8876 for PHP #70121
• CVE-2015-8877 for PHP #70064
• CVE-2015-8878 for PHP #70002
• CVE-2015-8879 for PHP #69975
• CVE-2015-8880 for PHP aa8cac57 (Dec 2015)

And while working on processing these issues for PHP, I also notice they weren’t updated for libGD where appropriate (including recent issues).

Beyond keeping the eco-system up to date, another aspect of publicity is getting opinions from other parties. For example, in the case of CVE-2015-8879, RedHat doesn’t agree with the classification of the bug as security. To give an example of a way things should happen is the request of CVE for PHP 5.5.34 in April, in which the Gentoo security team asked for assignment without upstream knowledge, Debian representative CCs upstream (Thanks!) and also asks CVE for issues covered by Ubuntu, to which the Ubuntu guy then adds some details.

I hope this blog post will reach the anonymous people behind these CVE requests, and also the people assigning them. Without transparency and keeping things in synchronization, the idea of having a centralized location for security warning is not going to accomplish its goals.

Backporting of PHP security fixes

4 months ago I wrote my thoughts about PHP support during the “PHP 5 support timeline” vote:

I think we should limit what we guarantee (meaning keeping only one year of security support till end of 2017), and encourage project members and the eco-system (e.g. Linux distributions) to maintain further security based on best effort.

This is already the case for out of official support releases like the 5.3 and 5.4 branches (examples for backports done by Debian: 5.3 and 5.4). And of course, we also have companies that make their money out of long term support (e.g. RedHat).

On the other hand, we should help the eco system in doing such extended support, and hosting backported fixes in the project’s git repo instead of having each Linux disto do the same patch work on its own.

But suggesting to others what they should do is easy, so I decided to finally find the time to also implement this myself. I’ve started with back porting PHP 5.5 fixes to PHP 5.4, resulting in a GitHub repository with all the fixes, including CVE info NEWS file entries and references to the original commits. See https://github.com/kaplanlior/php-src/commits/PHP-5.4-security-backports . I hope this would later on find it’s way into PHP LTS packages for Debian Wheezy.

Next step would be to start doing the same for PHP 5.3 (back porting from PHP 5.4, and later on also from PHP 5.5). This can be in use for RHEL 6.x (as LTS support for Debian Squeeze was recently finished).

The main idea of this repo, is to have a more central location for such work, hoping people would review and contribute fixes that should be taken into consideration.

During the process of digging into the CVE information and the commits, I’m also filling up a info such as CVE IDs to the NEWS file (e.g. https://github.com/php/php-src/pull/1892/files) and the web changelog (e.g. https://github.com/php/web-php/commits?author=kaplanlior), so users and researchers would find this info where it should be instead of digging themselves.

PHP 5 Support Timeline

With the new year starting the PHP project is being asked to decide about the PHP 5 support timeline.

While Aligning PHP 5.6 support timeline with the release date of PHP 7.0 seems like common sense to keep the support schedule continuous, there’s a big question whether to extend it further to an additional one year of security support till the end of 2018. This would make PHP 5.6, the last of the PHP 5 branch, to have 2 years of security support and de facto getting the same life span as PHP 7.0 would (ending support of both in Dec 2018).

But beside of support issues, this also affects adoption rate of PHP 7.0, serving as a catalyst due to end of support for the PHP 5 branch. My concerns are that with the additional security support the project would need to deal with the many branches (5.6, 7.0, 7.1, 7.2 and then to be 7.3 branch).

I think we should limit what we guarantee (meaning keeping only one year of security support till end of 2017), and encourage project members and the eco-system (e.g. Linux distributions) to maintain further security based on best effort.

This is already the case for out of official support releases like the 5.3 and 5.4 branches (examples for backports done by Debian: 5.3 and 5.4). And of course, we also have companies that make their money out of long term support (e.g. RedHat).

On the other hand, we should help the eco system in doing such extended support, and hosting backported fixes in the project’s git repo instead of having each Linux disto do the same patch work on its own.

CVE assignment without upstream knowledge

In the past few months I’ve been dealing with aligning PHP CVE information to enable easier tracking of security fixes. The two main locations are the NEWS file which is part of each release and the changelog available on the website which is more popular (and easier to update).

Usually the CVE are assigned per PHP.net security team request or with cooperation with one of the Linux distribution’s teams (either PHP or security), as should be in a good ecosystem.

Recently I got a few notifications issued by Debian about its PHP package, which I wasn’t familiar with these CVE IDS. When checking this, I found out a few CVE assigned per 3rd party (Linux distribution, bug reporter, etc…) request without upstream knowledge. Digging deeper I found out that some CVE were assigned a month after the fixes were released, while others were only a week or two after. While this makes sure the security information is documented, it’s harder to add the information after tagging and releasing.

In another case, while discussing about a CVE for a specific bug, we found out one was already assigned per the reporter’s request but without the our or the upstream library knowledge. Even if the issue isn’t severe, upstream should get a fair chance to fix issue before making them public. Which also leads to a problem with requesting CVE IDs on a public mailing list which in some cases leads to security information leakage. We should balance transparency with some grace period for upstreams (as projects share code).

PHP7 replaces non-free JSON extension

For many the PHP JSON extension license might look like a storm in a teacup, but for many Linux distributions the bits of the free software licenses are very important. That was the case when Debian decided (#692613) to remove the non-free JSON extension as part of the transition to PHP 5.5 in May 2013 (after the Debian 7 release).

This change was done with the help of Remi Collet (from Fedora / Red Hat) who wrote an alternative extension based on JSON-C implementation. A similar change was done by other Linux distributions, and this became the defacto standard for most Linux users.

The situation has recently changed with the acceptance of the Jakub Zelenka’s jsond RFC to replace the current non-free implementation with a free one. The change was committed to the code base on early February (Closing #63520) and expected to be released later this year as part of PHP7.

The importance of close integration between distribution and upstream

Many package maintainers need to decide when to upload a new version to Debian. Should the upload be done only after the official release, or is there a place for uploads during the development process. In the latter case there’s a need to balance between the benefit of early testing and feedback with the stability and not completely breaking user’s environment (and package relationships) too often.

With the coming PHP 5.6.0 release, Debian kept being on the cutting edge. Thanks to Ondřej, the new version was available in experimental since alpha1 and in unstable/testing since beta3. Considering the timing of the PHP release related to the Debian freeze, I’m happy we started early, and did the transition to PHP 5.6 a few months ago.

But just following the development releases (betas ,RCs) isn’t enough. Both Ondřej and myself are part of the PHP community, and know the planned timelines, current status and what are the critical points. Such knowledge was very useful this week, when we new 5.6.0 was pending finale tagging before release (after RC4). This made take the report of Debian bug #759381: “php5: TLS connections broken in 5.6.0 RC4” seriously and contact the release managers.

First it was a “heads up” and then a real problem. After a quick discussion (both private mails by me and on github by Ondřej), the relevant commit was reverted by the release managers (Julien Pauli & Ferenc Kovacs), and 5.6.0 was retagged. The issue will get more checks towards 5.6.1 without any time pressure.

Although 5.6.0 isn’t in production for anyone (yet), and like any major release can have issues, the close connectivity between everyone saved complaints from the PHP users and ecosystem. I don’t imagine the issue been sorted so quickly 16 hours later. This is also due to the bug been reported on difference between two close release (regression in RC4 comparing to RC3).

To close the loop, if we’ve uploaded 5.6.0 only after the release, the report would be regression between 5.5.x and 5.6.0, which is obviously much harder to pinpoint. So, I’m not sure I have a good answer for the question in the beginning of the post, but for this case our policy proved itself.