AGPL enforced: The Israeli ICT authority releases code

Data.gov.il was created in 2011 after the Israeli social justice protests as part of the the public participation initiative and started to offer data held by the government. Back then the website was based on Drupal. In 2016 it was changed to CKAN, a designated system for releasing data. This system is licensed under the AGPLv3 requiring source code availability for anyone who can access the the system over a network, de facto for every user.

Since the change to CKAN, open source people asked the state to release the code according to the license but didn’t get a clear answer. All this time when it’s clear it’s violation.  This led Gai Zomer to file a formal complaint in March 2017 with the Israeli State Comptroller. Absurdly, that same month the ICT authority mentioned a policy to release source code it owns, while failing to release code it has taken from others and adapted.

With the end of the summer break and Jew holidays, and after I wasn’t able to get the source, I decided to switch to legal channels, and with the help of Jonathan Klinger and my company, Kaplan Open Source Consulting, we notified they should provide the source code or we’ll address the court.

Well, it worked. In 3 days time the CKAN extensions where available on the website, but in a problematic way, so users weren’t able to download easily. This is why we decided not to publish this code release and let them fix it first. In addition we made it clear all the source code should be available, not only the extensions. Further more, if they already release it’s recommended to use git format instead of just “dumping” a tarball. So we told them if they aren’t going to make a git repository we’ll do that ourselves, but in any case, would prefer them to do that .

While this issue is still pending, the ICT authority had a conference called “the citizen 360” about e-gov and open government in which they reaffirmed their open source plans.

A slide about open source from the Israeli ICT authority presentation

Now, a month later, after our second letter to them, the about page in data.gov.il was updated with links to the ICT authority GitHub account which has the sources for the website and the extensions. A big improvement, and an important mark point as the commit to the repository was done by an official (gov.il) email address.

Beyond congratulating the Israeli ICT authority for their steps forward and the satisfaction of our insisting on them became fruitful, we would like to see the repository get updated on a regular basis, the code being given back to the various CKAN extensions (e.g. Hebrew translation). In general, we hope they would to get inspired by how the how data.gov.uk is doing technical transparency. If we allow ourselves to dream, we would like to see Israel becoming a dominate member in the CKAN community and among the other governments who use it.

We’re happy to be the catalyst for open source in the Israeli government, and we promise to keep insisted where needed. We know that due to other requests and notifications more organizations are on their way to release code.

(This post is a translation from Hebrew of a post in Kaplan Open Source Consulting at https://kaplanopensource.co.il/2017/11/20/data-gov-il-code-release/)

LibreOffice community celebrates 7th anniversary

The Document foundation blog have a post about LibreOffice 7th anniversary:

Berlin, September 28, 2017 – Today, the LibreOffice community celebrates the 7th anniversary of the leading free office suite, adopted by millions of users in every continent. Since 2010, there have been 14 major releases and dozens of minor ones, fulfilling the personal productivity needs of both individuals and enterprises, on Linux, macOS and Windows.

I wanted to take a moment to remind people that 7 years ago the community decided to make the de facto fork of OpenOffice.org official after life under Sun (and then Oracle) were problematic. From the very first hours the project showed its effectiveness. See my post about LibreOffice first steps. Not to mention what it achieved in the past 7 years.

This is still one of my favourite open source contributions, not because it was sophisticated or hard, but because it as about using the freedom part of the free software:
Replace hardcoded “product by Oracle” with “product by %OOOVENDOR”.

On a personal note, for me, after years of trying to help with OOo l10n for Hebrew and RTL support, things started to go forward in a reasonable pace, getting patches in after years of trying, having upstream fix some of the issues, and actually able doing the translation. We made it to 100% with LibreOffice 3.5.0 in February 2012 (something we should redo soon…).

FOSScamp Syros 2017 – day 3

The 3rd day should have started with a Debian sprint and then a LibreOffice one, taking advantage I’m still attending, as that’s my last day. But plans don’t always work out and we started 2 hours later. When everybody arrive we got everyone together for a short daily meeting (scrum style). The people were divided to 3 teams for translating:  Debian Installer, LibreOffice and Gnome. For each team we did a short list of what left and with what to start. And in the end – how does what so there will be no toe stepping. I was really proud with this and felt it was time well spent.

The current translation percentage for Albanian in LibreOffice is 60%. So my recommendation to the team is translate master only and do not touch the help translation. My plans ahead would be to improve the translation as much as possible for LibreOffice 6.0 and near the branching point (Set to November 20th by the release schedule) decide if it’s doable for the 6.0 life time or to set the goal at 6.1. In the 2nd case, we might try to backport translation back to 6.0.

For the translation itself, I’ve mentioned to the team about KeyID language pack and referred them to the nightly builds. These tools should help with keeping the translation quality high.

For the Debian team, after deciding who works on what, I’ve asked Silva to do review for the others, as doing it myself started to take more and more of my time. It’s also good that the reviewer know the target language and not like me, can catch more the syntax only mistakes. Another point, as she’s available more easily to the team while I’m leaving soon, so I hope this role of reviewer will stay as part of the team.

With the time left I mostly worked on my own tasks, which were packaging the Albanian dictionary, resulting in https://packages.debian.org/sid/myspell-sq and making sure the dictionary is also part of LibreOffice resulting in https://gerrit.libreoffice.org/#/c/41906/ . When it is accepted, I want to upload it to the LibreOffice repository so all users can download and use the dictionary.

During the voyage home (ferry, bus, plain and train), I mailed Sergio Durigan Junior, my NM applicant, with a set of questions. My first action as an AM (:

Overall FOSScamp results for Albanian translation were very close to the goal I set (100%):

• Albanian (sq) level1 – 99%
• Albanian (sq) level2 – 25% (the rest is pending at #874497)
• Albanian (sq) level3 – 100%

That’s the result of work by Silva Arapi, Eva Vranici, Redon Skikuli, Anisa Kuci and Nafie Shehu.

FOSScamp Syros 2017 – day 2

The morning stated by taking the bus to Kini beach. After some to enjoy the water (which were still cold in the morning), we sat for talking about the local Debian community and how can we help it grow. The main topic was localization (l10n), but we soon started to check other options. I reminded them that l10n isn’t only translation and we also talked about dictionaries for spell checking, fonts and local software which might be relevant (e.g. hdate for the Jewish/Hebrew calendar or Jcal for the Jalali calendar). For example it seems that regular Latin fonts are missing two Albanian characters.

We also talked about how to use Open Labs to better work together with two hats – member of the local FOSS community and also as members of various open source projects (not forgetting open content / data ones projects as well). So people can cooperate both on the local level, the international level or to mix (using the other’s project international resources). In short: connections, connections, connections.

Another aspect I tried to push the guys toward is cooperating with local companies about open source, whether it’s the local market, the municipal and general government. Such cooperation can take many forms, sponsoring events / giving resources (computers, physical space or employee’s time) and of course creating more jobs for open source people, which in turn will support more people doing open source for longer period.

One of the guys thought  benefit the local community will benefit from a mirror server, but that also requires to see the network topology of Albania to make sure it makes sense to invest in one (resources and effort).

We continued to how it would be best to contribute to open source, mostly that Debian, although great isn’t always the best target, and they should always try to work with the relevant upstream. It’s better to translate gnome upstream then sending the Debian maintainer the translation to be included in the package. That shortcut can work if there’s something urgent like a really problematic typo or something what unless done before the release would require a long long wait (e.g. the next Debian release). I gave an example that for important RTL bugs in LibreOffice I’ve asked Rene Engelhard to include the patch instead of waiting for the next release and its inclusion in Debian.

When I started the conversation I mentioned that we have 33% females out of the 12 participants. And that’s considered good comparing to other computer/technical events, especially open source. To my surprise the guys told me that in the Open Labs hackerspace the situation is the opposite, they have more female members than male (14 female to 12 male). Also in their last OSCAL event they had 220 women and 100 men. I think there’s grounds to learn what happens there, as the gals do something damn right over there. Maybe Outreachy rules for Albania should be different (:

Later that day I did another session with Redon Skikuli to be more practical, so I started to search on an Albanian dictionary for spell checking, found an old one and asked Redon to check the current status with the guy. And also check info about such technical stuff with Social Sciences and Albanological Section of the Academy of Sciences of Albania, who is officially the regulator for Albanian.

In parallel I started to check how to include the dictionary in LibreOffice, and asked Rene Engelhard to enable Albanian language pack in Debian (as upstream already provide one). Checking the dictionaries I’ve took the opportunity to update the Hebrew. It took me a little longer as I needed to get rust off my LibreOffice repositories (dictionaries is a different repository) and also the gerrit setup. But in the end: https://gerrit.libreoffice.org/#/c/41864/

With the talks toady and the starting to combine both Debian and LibreOffice work today (although much of it was talking) – I felt like I’m the right person on the right place. I’m happy to be here and contribute to two projects in parallel (:

The commit police: learning for recent reference mistakes

Continuing the previous post about commits and bugs, I’d like to review some mistakes I saw recently. Mistakes do happen, but mentioning them here is meant to teach others and hopefully to reduce similar ones in the future. This post isn’t about shaming the authors/commiters. Also, the points I highlight are what I consider as a mistakes or problems, other people might think differently.

1. Mentioning two bugs in one commit message, which our system doesn’t support right now. So the second bug doesn’t get the commit notification, and that should be done manually (example: commit 2933935 and fdo#53278).
2. Referencing gerrit changes as part of the commit message (example: commit 87f185d). Giving references as part of the commit is great and helpful, but I would prefer to see the reference to the actual commit and not to its review process. This is meaningful when you search the log. If the followup change is suggested when the first change is still in review it should be combined, otherwise it already have a commit to reference.
3. Following up a commit, and not mentioning the bug it references (example: commit 87f185d). In this specific case, the we’re talking about a meta bug for translating comments from German to English (fdo#39468), so no harm done. But this important when you want to cherry pick a fix for a bug to other branches and might forget the follow up commits. It’s also relevant to more technical meta bugs (example: fdo#62096).
4. Referencing a mailing list which reference a commit and a bug instead of referencing them directly (commit 21fea27, fdo#60534). This bug shows very well why correct referencing is important, a commit was made to fix the bug, a follow up commit was done without proper reference, and than the first commit was reverted. No one involved in trying to fix that specific bug knew about the follow up commit as it wasn’t documented anywhere.
5. Referencing bugs though full bug URL instead of the right format (example: commit e1f6dac). Also the bug is referenced in the commit in the body of the message instead of the first line (header) which is more visible.
6. Referencing non existing bugs (example: commit 3a4534b). Which got a manual notification in the bug by the comitter (fdo#33091).
7. Using shortening services URL as part of commit messages (example: commit 86f8fba). There’s no way to know to what the reference is without using the service, which in this case was leading to a post on one of the projects mailing list. There isn’t any problem giving the direct URL to the list’s archive. It’s interesting whether we should link to our URL and is it “OK” to use other external services who also archive our mailing lists (example: commit e83990a).

To conclude, having references in commit message is really helpful, but please reference the right resource to help people find it easily and to let our automated services parse it.

The commit police: reference you bugs properly

Besides my RTL work on Libreoffice, every once in a while I just go over the regular commit log to see what have changed. I don’t necessary understand each line in the commit, but do get the general idea from the commit message. Being more dependent on the commit messages makes me review them more thoroughly (hence the topic of this post).

As many projects, Libreoffice has notifications of commits related to bugs reports when the bug number is properly mentioned in the commit message. This is very useful for other developers and also for QA people. After verification of a bug is fixed, I often use the commits listed on the bug report to cherry pick them to an older branch. Going to search for an unreferenced commit isn’t much fun.

One of the things I notice is different ways people reference bugs – from not referencing them at all to referencing them in various ways like linking to the bugs system, just writing the number (without the fdo# prefix) and other creative ways… This is also true for first time contributors, which might not know the standards or the “rules”. When I see such a case I usually put a manual notification in the bug report, and mail the author of doing so. For new or sporadic contributors this is also an opportunity to welcome and thank them for the commit and even encourage future contributions.

I have been asked why aren’t that info on the wiki, so I went looking and found out the info is in the right place under the “Development/GetInvolved” page. The relevant part is:

When you type a commit message:

• start the first line with a bug reference like fdo#12345, if you have one for your commit (see details below)
• use the rest of the first line as a concise summary of your changes
• the 2nd line remains empty
• and starting on the 3rd line you can explain how and what changes have been made for what reasons.

Thanks in advance and happy coding (:

Team work on RTL bugs

After returning from a vacation, I went over my RTL bugs backlog. I missed helping with checking a few patches, so I got left with verifying the fixes done by the KACST guys on master. Last week was dedicated for verifications of the bug fixes, even found a regression and reverted the commit after discussion with the author.

Yesterday, I took my time with pushing these fixes also to the 4-0 branch, having them available as soon as possible instead of waiting for the 4.1 release. Total of 7 bugs got pushed, see https://wiki.documentfoundation.org/RTL_Bugs#4.0.3 for the list. I notified the developers, so they’ll know their changes are available to the public sooner than expected (and let the people enjoy their work earlier). All these pushes make sure that master doesn’t have any RTL support superiority over the current stable branch.

In one of the cases I had to do some follow up about the commit, finding another relevant commit. This was merged with a trick Eike taught me a while ago for squashing commits with git rebase. Fridrich quickly helped with verifying that the squash stills looks good, letting me push another dependent fix on top of it.

RTL status for Libre Office 4.0.0

LibreOffice 4.0.0 will be released soon, and it’s a good time to summarize its RTL status. Similar to 3.5.0, the RTL work started with LibreOffice conference in October, showing the current state (see LibreOffice RTL status 2012 presentation) and listing the top problems. The main goal of this talk is to raise awareness to RTL issues and recruit developer to help fix them. This was exactly what happened with fdo#44657 which is about Calc not having a scroll bar in RTL user interface. This way RTL bug #1 for almost a year.

In total, version 4.0.0 has 8 bugs fixed (mainly Calc, Presenter Console and toolbars). In addition it has 5 language specific bugs fixed. Full lists are at https://wiki.documentfoundation.org/RTL_Bugs#4.0.0 and https://wiki.documentfoundation.org/RTL_Bugs#4.0.0_2. Beside working on fixing bugs for the 4.0.0, another focus was put on making sure the fixes are available for the 3.6.x users. This resulted with cherry picking 11 out of 13 bug fixes (part of 3.6.4 and 3.6.5 releases). More fixes are already underway in the master branch, and will also be part of 4.0.1 maintenance release (full list at https://wiki.documentfoundation.org/RTL_Bugs#4.0.1).

Beside thanking the core developers  for their help (patches, code review), I’d like to thank the guys from The National Program for Free and Open Source Software Technologies (Motah) at King Abdulaziz City for Science and Technology (KACST) in Saudi Arabia, which have joined the RTL team during this release, and their effect is well noticed.

Despite all the work done, we still have a share of RTL bugs for the next release, these are the 7 top priority RTL bugs, any help would be appreciated.

1. fdo#33302 Brackets inverted in rtl text (mac only)
2. fdo#53030 Calc deletes current sheet, not right-clicked on sheet when trying to delete opposite directionality sheets
3. fdo#56403 RTL UI: with RTL locales presentations appear with text align to the right instead of to the left
4. fdo#37128 Writer saves text alignment of RTL paragraph not according to the ODF specification
5. fdo#44029 docx files with RTL paragraphs aren’t compatible with Word
6. fdo#43093 FILEOPEN: [docx] MS Office documents opens with wrong alignment and directionality
7. fdo#56408 Brackets are not handled correctly with mixed English/Latin and Hebrew/Arabic texts

LibreOffice conference and motivation

Last week, just before the branching of 3.6.4 was the peak of my recent work in LibreOffice. The motivation from attending the conference and working with others lasted long for me with the help of a few collaborations done since (thanks Caolan and Eike). My own patches were approved just in time to make it to the 3-6 branch, which, along other fixes, pretty much makes 3.6.4 a better release to the RTL users in general, and Hebrew users specifically. Things aren’t perfect, but they’re improving nicely (on going status is at RTL bugs wiki page).

I’m also very happy that most RTL related fixes since the conference have been cherry-picked from master to 3-6 branch, so the users won’t have to wait till February for the 4.0 release. Last year I had to wait till the 3.5 release to see some of my work fruits, now I can see them in much shorter intervals.

The motivation and involvement go hand in hand, each enhances the other for me. I’ve started reviewing patches sent for RTL bugs to verify the suggested fixes with private builds and marking bugs as verified when the patches do get accepted. Compering to a month ago, I’m now busy with helping the developers do their part instead of “just” reporting problems. Seeing things go forward (and helping them do so) is much more motivating than waiting for someone to pick the bug report and try solve it.

I hope having fast response to their code changes is also motivating for the developers, in that case everyone enjoys the collaboration.

How I got an extra value from my flight ticket to libreoffice 2011

Miklos published about his “LibreOffice RTF import Drawing Objects improvements” which was done to fix my report (fdo#42407). As an anecdote I want to mention that he test document was my flight ticket to the LibreOffice conference in 2011.

So I feel I got an extra value from attending the conference in the form of having a this RTF compatibility implemented. In addition, Miklos fixed an import problem with Hebrew with these old documents (fdo#56512) – double value. Miklos – I owe you a drink (:

I’m not sure if I should thank the traveling company for working with such old software which produced the old formatted RTF files, but in any case, it turned out to be a good thing.