summaryrefslogtreecommitdiff
path: root/KEP-0014.txt
blob: 2d8f1f69cd6a229f4fb7cb712c3b5b3c8e10f124 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
{{kep
 |number=14
 |ticketnumber=
 |title=Non-conflicting edits of RFC5228/Sieve scripts by multiple editors
 |author=Georg Greve
 |author_email=greve@kolabsys.com
 |status=draft
 |type=design
 |creation_date=
 |obsoleted_by=
 |related=
}}


== Abstract ==

This Kolab Enhancement Proposal (KEP) <ref name="kep">[[KEP:1]] Bootstrapping the KEP process</ref> defines the conventions and application behaviour to enable non-conflicting edits of {{rfc|5228}}<ref name="sieve">{{rfc|5228|title=Sieve: An Email Filtering Language}}</ref> Sieve scripts on the server by multiple editors. It follows the basic ideas outlined in an email to the Kolab development mailing list in May 2011 <ref name="multisieve">Georg Greve: [http://kolab.org/pipermail/kolab-devel/2011-May/012846.html Enabling multiple sieve editors on Kolab installation]</ref> by making use of the 'include' functionality defined in the 'Sieve Email Filtering: Include Extension' draft<ref name="include">[https://datatracker.ietf.org/doc/draft-ietf-sieve-include/ Sieve Email Filtering: Include Extension]</ref>.

The result is the ability to have different layers of Sieve management per user, with different levels of required authorization, and the ability to provide users with multiple active scripts which can be activated and deactivated as required, thus enabling splitting of the user-facing Sieve functionality into more conveniently managed files. For more information on what this KEP does or does not address, please see the [[#Rationale|Rationale]] section and its [[#Scope_of_this_KEP|Scope of this KEP]] subsection, in particular.

== Naming Conventions ==
For the purpose of this KEP, all file names of Sieve<ref name="sieve" /> scripts '''SHALL''' be given in their 'basename' only, so without extension, which is customarily dependent upon the specific IMAP server used. Cyrus IMAP commonly uses '.siv', Dovecot commonly uses '.sieve', other servers or variations of these servers may use different conventions. These extensions '''MAY''' be added as required or customary for the particular installation, but only the basename '''SHALL''' be considered authoritative.

For reading and writing, clients '''SHALL''' consistently use the upper case form for all of [[#MASTER|MASTER]], [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] and '''SHALL''' append extensions in lower case, e.g. '.siv' and '.sieve' as done consistently in this KEP.

== Enabling Non-Conflicting Sieve Editing ==
In order to enable Non-Conflicting Sieve Editing according to this KEP, a [[#MASTER|MASTER]] file '''MUST''' be defined during platform integration and adapted during deployment. The [[#MASTER|MASTER]] '''SHALL''' then be considered authoritative for that deployment. 

Where the IMAP server supports at least version 13 of the 'Sieve Email Filtering: Include Extension' draft<ref name="include" />, the [[#MASTER|MASTER]] file '''SHOULD''' use the ':optional' parameter for its includes and '''SHALL''' constitute the entire authoritative set of files for this deployment. Otherwise the [[#MASTER|MASTER]] file '''MUST''' be accompanied by a no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] file which '''SHALL''' constitute part of the authoritative set of files for this deployment alongside the [[#MASTER|MASTER]] file.

The authoritative set of files '''SHALL''' be automatically deployed and activated for all users in that system that are to make use of the Non-Conflicting Sieve Editing alongside the no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] files.

{{note|Why and when do we need dummy files?|The 'Sieve Email Filtering: Include Extension' draft up until version 12 specifies that the server MUST generate an error when trying to include non-existing files. In such cases, because [[#MASTER|MASTER]] is immediately active, typically before the management system or the user had any time to touch the Sieve scripts, empty files are required to avoid superfluous error messages and potentially break the functionality, e.g. when the user has created Sieve scripts, but the management system has not yet done so. From version 13 of the draft onwards the ':optional' parameter can be used to avoid this issue, but this KEP cannot rely upon all implementations being against version 13 or later.}}

=== Sieve Editor Requirements ===
To be compliant with this KEP, Sieve editors '''MUST''' check for the presence of the [[#MASTER|MASTER]] file, and, if present, '''MUST''' do the following:
* hide [[#MASTER|MASTER]] and all other [[#Protected Names|Protected Names]] from user selection;
* protect all [[#Protected Names|Protected Names]] by preventing users from choosing them for their own scripts;
* define a consistent name and version to be used in the provisions outlined in [[#User_specific_scripts|User specific scripts]];
* upon startup read [[#USER|USER]] to get the number and order of active scripts;
* regularly check for changes by other editors to [[#USER|USER]] or any of the active scripts;
* edit scripts in line with the [[#User_specific_scripts|User specific scripts]] provisions;
* overwrite [[#USER|USER]] to reflect activations/deactivations of [[#User_specific_scripts|User specific scripts]] by including/excluding them from the [[#USER|USER]] script;
* ensure that after finishing its modification(s), [[#MASTER|MASTER]] is the active script.

At the choice of its author, a Sieve editor '''MAY''' visualize scripts that are executed before or after the scripts edited by the user. How this is implemented is largely up to the individual editors, but it should ensure that
* it is made clear to the user where appropriate that these rules cannot be edited within this editor, e.g. through graying out;
* the user is given some indication as to where these rules can be edited, e.g. in the management console.

If there is no [[#MASTER|MASTER]] file in the users list of Sieve scripts, editors '''MAY''' behave as they see fit, but '''MUST''' always ensure its users do not use [[#Protected Names|Protected Names]] and '''MUST''' always ensure that users do not create a [[#MASTER|MASTER]] which would trigger the functionality according to this KEP.

{{warning|Nesting depth|The nesting depth for inclusion will typically be limited by the server, with a minimum of three levels, as specified by the 'Sieve Email Filtering: Include Extension' draft<ref name="include" />. The mechanisms in this KEP use two of these levels for management purposes, so it is recommended that editors do not provide for include functionality in their user scripts unless they know the server to support deeper nesting. The maximum level of supported nesting is the ''maximum depth supported by the server minus two''.}}

=== Management system requirements ===
To be compliant with this KEP, management systems that make use of Sieve for some of their functionality '''MUST''' check for the presence of the [[#MASTER|MASTER]] file, and, if present, '''MUST''' do the following:
* limit themselves to the [[#MANAGEMENT|MANAGEMENT]] script;
* define a consistent name and version to be used in the provisions outlined in [[#MANAGEMENT|MANAGEMENT]];
* write out the [[#MANAGEMENT|MANAGEMENT]] script as defined.

The [[#MANAGEMENT|MANAGEMENT]] script '''SHALL NOT''' include any script from the user namespace, it '''MAY''' only include files from the global namespace.

If there is no [[#MASTER|MASTER]] file in the users list of Sieve scripts, management systems '''MAY''' behave as they see fit, but '''MUST''' always ensure its users do not use [[#Protected Names|Protected Names]].

If the management system aims to also control (de-)activation of the functionality defined in this KEP through creation or deletion of the [[#MASTER|MASTER]] script, it '''MUST''' ensure that it writes the [[#MASTER|MASTER]] as defined authoritatively for this platform or group of users, and '''MUST''' always write the [[#MANAGEMENT|MANAGEMENT]] script as defined herein.

Sticking to the conventions for [[#MANAGEMENT|MANAGEMENT]] scripts is recommended for all management systems in order to facilitate easier deployment/integration with the Kolab Groupware Solution.

=== Integration Requirements ===
Integrators and Kolab Certified Professionals responsible for the deployment '''SHALL''' ensure that if the platform or deployment intends to make use of this feature, users which are supposed to make use of this feature will have their [[#MASTER|MASTER]] file created and activated upon account creation alongside the no-op [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] files.

This can be in the form of adding this to the default behaviour for all users, by tying this to some role or group in LDAP, or whichever other approach the platform takes on configuration.

Sieve editors '''MUST''' be able to rely on the proper creation of the [[#MASTER|MASTER]] script before they are called for the first time. It is the responsibility of the platform integrator or deployment engineer to ensure this is done properly.

{{note|Limited number of sieve scripts|IMAP servers typically limit the number of sieve scripts a user can store. A common value for this number would be five. This KEP uses three scripts in the users name space for special purposes, so a normal IMAP server might only allow two more scripts defined by the user themselves. It is therefore suggested to raise the number of scripts by three to keep the number of total user defined scripts constant, or whatever seems sensible for this particular platform or installation.}}

=== MASTER ===
This is an example of the MASTER script, which can act as a template and '''SHALL''' be modified during integration and deployment as required for the particular installation. 

 #
 # MASTER
 # 
 # This file is authoritative for your system and MUST BE KEPT ACTIVE.
 #
 # Altering it is likely to render your account dysfunctional and may
 # be violating your organizational or corporate policies.
 # 
 # For more information on the mechanism and the conventions behind
 # this script, see http://wiki.kolab.org/KEP:14
 #
 
 require ["include"];
 
 # OPTIONAL: Includes for all or a group of users
 # include :global "all-users";
 # include :global "this-group-of-users";
 
 # The script maintained by the general management system
 include :personal :optional "MANAGEMENT";
 
 # The script(s) maintained by one or more editors available to the user
 include :personal :optional "USER";

MASTER, [[#MANAGEMENT|MANAGEMENT]] and [[#USER|USER]] '''SHALL''' be receiving their file extension as required and appropriate for the IMAP server used.

MANAGEMENT is the script written by the management system of the platform, USER is the script intended for controling the execution of individual scripts by a user. General sieve editors available to users and platform management systems '''MUST NOT''' alter this script.

=== MANAGEMENT ===
Management systems '''SHALL''' define the variables 'MANAGEMENT_SYSTEM' and 'MANAGEMENT_SYSTEM_VERSION' through comments '# MANAGEMENT_SYSTEM: <value>' and '# MANAGEMENT_SYSTEM_VERSION: <value>' in the comment block above the first Sieve script commands. 

Otherwise management systems are free to write to each user's MANAGEMENT script as required and appropriate.

General sieve editors available to users '''MUST NOT''' alter this script.

=== USER ===
Editors '''SHALL''' be reading in the USER script, parsing it for individual includes of user-defined scripts, which take the form of

  include :personal "<filename>"

where <filename> is the file name of a local script of the current user.

Parsing '''SHALL''' preserve all comments that do not conform to the above format and discard the rest of the file.

When replacing the USER script on the server, editor '''SHALL''' first write out all preserved comments, followed by an empty line, then the include requirement, followed by an empty line, and then iterate over all client side includes, each followed by an empty line. 

The order of entries '''SHALL''' always be preserved unless explicitly changed by the user.

==== Sample USER Script ====

  # USER Management Script
  #
  # This script includes the various active sieve scripts
  # it is AUTOMATICALLY GENERATED. DO NOT EDIT MANUALLY!
  # 
  # For more information, see http://wiki.kolab.org/KEP:14#USER
  #
 
  require ["include"];
  
  include :personal "superspamrules"
  
  include :personal "mailinglists"
  
  include :personal "personalmail"
  
  include :personal "workmail"
  

=== User specific scripts ===
User specific scripts contain the actual Sieve commands as defined by the user through the editors. 

When creating a new script, editor '''SHALL''' define the variables 'EDITOR' and 'EDITOR_VERSION' through comments '# EDITOR: <value>' and '# EDITOR_VERSION: <value>' in the comment block above the first Sieve script commands. 

When parsing an existing script, editor '''SHALL''' parse for the variables 'EDITOR' and 'EDITOR_VERSION' and based on their values behave as follows:
* '''Same editor, same version'''
: If the variables are set and match their name and version, editor '''SHALL''' parse the script as usual and present it to the user for display/editing as required.
* '''Same editor, different version'''
:If the variables are set and match the name of the editor, but with different version, editor '''SHALL''' attempt to parse the script and upon success present it to the user for display/editing as required, saving it out with its own version. If editor cannot parse the script, it '''SHALL''' inform user about the version incompatibility and '''MAY''' choose to display the script in raw text form, as best it can, or not at all and '''SHALL''' always consider it '''read only'''.
* '''Different, supported editor'''
:If the variables are set and match a different editor and version which is supported by the editor reading the script, editor '''MAY''' parse the script normally and present it to the user for display/editing. When changing the file on the server, editor '''MAY''' offer the user to take ownership of the script, '''SHALL''' require postitive confirmation from the user for doing so, and '''MUST''' then write out the script in its own format and with the editor and version variables set to its own name and version. Otherwise editor '''SHALL''' preserve the format of the editor and version which originally wrote the file and '''MUST''' preserve the variables accordingly. 
* '''Different editor'''
:If the variables are set and match a different editor, editor '''MAY''' attempt to parse the script. Upon success, editor '''SHALL''' default to presenting the user with the script for display in '''read-only''' mode. Editor '''MAY''' allow the user to edit the script, '''SHALL''' require postitive confirmation from the user for taking ownership by this editor, and '''MUST''' then write out the script in its own format and with the editor and version variables set to its own name and version.
:If editor has not attempted to parse the script, or could not successfuly parse, it '''SHALL''' treat the script as read-only and '''MAY''' present it to the user as best it can, in raw format, or not at all.
* '''Variables not set'''
:If the variables are '''not''' set, editor '''SHALL''' assume a manually maintained script. It '''MAY''' attempt to parse the script and proceed as outlined in the 'Different editor' provisions, above.
* '''Variables set to READONLY, NONE'''
:If the editor variable is set to 'READONLY' and the version is set to 'NONE', editor '''SHALL''' treat the script as '''explicitly read only'''. It '''MAY''' attempt to parse and display, but it '''SHALL NOT''' modify this script.

==== Other provisions ====
For IMAP servers which support {{rfc|2342}}<ref name="namespace1">{{rfc|2342|title=IMAP4 Namespace}}</ref>/{{rfc|4466}}<ref name="namespace2">{{rfc|2342|title=Collected Extensions to IMAP4 ABNF}}</ref> clients '''SHOULD''' make use of the NAMESPACE command to retrieve the values of the folder hierarchy delimiter or separator, and the prefixes for personal, shared and (other) user folders upon reading and writing sieve scripts to the server to ensure folders are correctly mapped.

Editors '''MAY''' also want to check commands against IMAP ACLs to prevent creation of illegal rules at the time of editing the script, instead of during execution time.

Editors '''MAY''' also want to take into account that not all IMAP servers support {{rfc|5232}}<ref name="sieve">{{rfc|5232|title=Sieve Email Filtering: Imap4flags Extension}}</ref> yet, but implement earlier draft versions of what became the RFC.

==== Sample user specific script ====

A manual script example:

  # Spam Rules, manually defined
  require ["fileinto", "comparator-i;ascii-numeric", "imapflags", "regex"];
 
  if anyof ( not exists ["From","Date"],
           header :comparator "i;ascii-casemap" :contains "X-Spam-Level" "*****",
           header :is ["X-Spam-Flag", "X-Spam-Status"] ["YES","Yes","yes"] ) {
    setflag "\\Seen";
    fileinto "INBOX/Spam";
    stop;
  }

The same script as an explicit '''read only, do not touch''' script:

  # Spam Rules, manually defined
  # EDITOR: READONLY
  # EDITOR_VERSION: NONE
  require ["fileinto", "comparator-i;ascii-numeric", "imapflags", "regex"];
 
  if anyof ( not exists ["From","Date"],
           header :comparator "i;ascii-casemap" :contains "X-Spam-Level" "*****",
           header :is ["X-Spam-Flag", "X-Spam-Status"] ["YES","Yes","yes"] ) {
    setflag "\\Seen";
    fileinto "INBOX/Spam";
    stop;
  }

=== Protected Names ===
All protected names '''SHALL''' be protected in their basename and all extensions, e.g. the listing of MASTER in the table below '''MUST''' be read to also protect MASTER.sieve, MASTER.siv explicitly, and more generally MASTER.*.

Protected names under this KEP '''SHALL''' be
* 'MASTER' for the primary script, which '''SHALL''' be defined at integration stage authoritatively for all users and kept active for each user at all times;
* 'MANAGEMENT' for the script written out by management system of the system;
* 'USER' for the control script of user editable scripts.

All considerations of protected names '''SHALL''' be case-insensitive, e.g. MASTER, master, mAster and Master are '''all''' considered protected names regardless of their extension (if any). 

== Upgrade Path ==

This KEP introduces new conventions. 

Older clients must be expected to ignore these conventions, consequently breaking the functionality in ways that will be undesirable to the user, but should normally not result in data loss.

=== Updates of 'Sieve Email Filtering: Include Extension' draft ===

This KEP '''SHALL''' remain authoritative for the latest version of the latest draft of 'Sieve Email Filtering: Include Extension' draft<ref name="include" /> and ultimately the resulting RFC, provided no revision introduces functionality incompatible with this KEP, in which case the ''last compatible version'' '''SHALL''' be authoritative until this KEP can be obsoleted and replaced by a subsequent Kolab Enhancement Proposal.

== Rationale ==
There are multiple issues around Sieve (RFC 5228<ref name="sieve" />) scripts. This KEP will provide a workaround for '''some''' of these issues, but it cannot address all of them.

=== Issue No 1 ===
A major issue is the incompatibility of multiple Sieve editors with each others' output. Due to the complexity of Sieve, editors typically only implement a subset of how certain rules are expressed, typically oriented around how the editing of these rules is presented to the user. When presented with an arbitrary Sieve script written by another editor or human being, editors typically '''CANNOT''' just parse this script and present it visually for editing.

Reasons for this are the complexity and flexibility of the language, which make it hard for clients to parse existing scripts into their internal model and representation of what kind of filtering is supposed to take place how and when. To circumnavigate this issue and allow users to edit existing scripts, editors typically have some metainformation about the rules that have been expressed in Sieve which are modeled after their own approach to editing these scripts. That information is either stored in a separate data store, such as the file system or an SQL based data base, or injected into the Sieve script itself as comments.

This fundamental incompatibility is the reason that it is very difficult for two different Sieve editors to work on the same script, and the reason why usually only one Sieve editor is authoritative in any one installation. 

Providing the first step of resolving this issue at the root through a joint XML based meta language is the intent of {{rfc|5784}}<ref name="sievexml">{{rfc|5784|title=Sieve Email Filtering: Sieves and Display Directives in XML}}</ref>. In order to implement this there would have to be a subsequent definition of metadata fields, so no Sieve editor currently supports this.

=== Subsequent Issue ===
The Kolab Groupware Server often finds itself integrated into existing products which provide their own user and system administration infrastructure. This infrastructure typically also involves some level of vacation notice and/or spam management which may be the preferred way for customers to make use of such functionality. 

That functionality is typically implemented through Sieve scripts. But because there can ever only be one active Sieve script, and because editors don't mix, this often means that installations have to disable all Sieve based filtering for their users in order to protect the managed functionality.

=== Issue No 2 ===
With the Sieve: Include Extension<ref name="include" />, Sieve knows two namespaces, a global one shared among all users, and a local one for each user. Global scripts can be included locally, but are never executed automatically, while local scripts are always editable by the user.

For corporate environments, this is not always desirable. Admins may want to set scripts from the global context to be executed or even have user specific scripts that are being executed in a way that does not allow the user to accidentally or intentionally disable these scripts. The inability of Sieve to protect local scripts from editing, or define global scripts that are always executed then results in a similar workaround to the one described in the [[#Subsequent_Issue|Subsequent Issue]], above: Access to Sieve is blocked in its entirety for all users.

It should be noted that the 'Sieve -- Sequential Execution of Multiple Scripts' draft<ref name="multiscript">[https://tools.ietf.org/html/draft-degener-sieve-multiscript-00 Sieve -- Sequential Execution of Multiple Scripts]</ref> sought to provide this functionality through an extension of Sieve itself, but expired in October 2003 and never made it to RFC stage. Should this be picked up again and moved forward, it would provide a more robust solution to this issue and Kolab should probably move towards it.

=== Issue No 3 ===
Sieve scripts can get fairly complex. It would be much more convenient and easier to manage the functionality with different scripts that define certain subsets of functionality and get executed as required. This was the original intention behind the 'Sieve Email Filtering: Include Extension' draft<ref name="include" />.

=== Scope of this KEP ===
This KEP addresses the [[#Subsequent_Issue|Subsequent Issue]] and [[#Issue_No_2|Issue No 2]] through a workaround that is largely based on convention and procedure in a way that keeps the resolution of [[#Issue_No_3|Issue No 3]] through the Include Extensiont<ref name="include" /> intact. It does nothing to address [[#Issue_No_1|Issue No 1]], which is out of its reach, although hopefully the workaround may help to bring some convergence in the field of editors and consequently contribute to a solution for that issue some time in the future, ideally based upon the work done in {{rfc|5784}}<ref name="sievexml"/>.

== References ==

{{Reflist}}

== Copyright ==

This document has been placed in the public domain.