forked from celery/celery
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathChangelog
1605 lines (985 loc) · 48 KB
/
Changelog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.. _changelog:
================
Change history
================
.. contents::
:local:
If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
.. _version-3.0.25:
3.0.25
======
:release-date: 2014-07-10 05:00 P.M UTC
Security Fixes
--------------
* [Security: `CELERYSA-0002`_] Insecure default umask.
The built-in utility used to daemonize the Celery worker service sets
an insecure umask by default (umask 0).
This means that any files or directories created by the worker will
end up having world-writable permissions.
Special thanks to Red Hat for originally discovering and reporting the
issue!
This version will no longer set a default umask by default, so if unset
the umask of the parent process will be used.
.. _`CELERYSA-0002`:
http://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0002.txt
.. _version-3.0.24:
3.0.24
======
:release-date: 2013-10-11 04:40 P.M BST
- Now depends on :ref:`Kombu 2.5.15 <kombu:version-2.5.15>`.
- Now depends on :mod:`billiard` version 2.7.3.34.
- AMQP Result backend: No longer caches queue declarations.
The queues created by the AMQP result backend are always unique,
so caching the declarations caused a slow memory leak.
- Worker: Fixed crash when hostname contained Unicode characters.
Contributed by Daodao.
- The worker would no longer start if the `-P solo` pool was selected
(Issue #1548).
- Redis/Cache result backends would not complete chords
if any of the tasks were retried (Issue #1401).
- Task decorator is no longer lazy if app is finalized.
- AsyncResult: Fixed bug with ``copy(AsyncResult)`` when no
``current_app`` available.
- ResultSet: Now properly propagates app when passed string id's.
- Loader now ignores :envvar:`CELERY_CONFIG_MODULE` if value is empty string.
- Fixed race condition in Proxy object where it tried to
delete an attribute twice, resulting in :exc:`AttributeError`.
- Task methods now works with the :setting:`CELERY_ALWAYS_EAGER` setting
(Issue #1478).
- :class:`~kombu.common.Broadcast` queues were accidentally declared
when publishing tasks (Issue #1540).
- New :envvar:`C_FAKEFORK` environment variable can be used to
debug the init scripts.
Setting this will skip the daemonization step so that errors
printed to stderr after standard outs are closed can be seen::
$ C_FAKEFORK /etc/init.d/celeryd start
This works with the `celery multi` command in general.
- ``get_pickleable_etype`` did not always return a value (Issue #1556).
- Fixed bug where ``app.GroupResult.restore`` would fall back to the default
app.
- Fixed rare bug where built-in tasks would use the current_app.
- :func:`~celery.platforms.maybe_fileno` now handles :exc:`ValueError`.
.. _version-3.0.23:
3.0.23
======
:release-date: 2013-09-02 01:00 P.M BST
- Now depends on :ref:`Kombu 2.5.14 <kombu:version-2.5.14>`.
- ``send_task`` did not honor ``link`` and ``link_error`` arguments.
This had the side effect of chains not calling unregistered tasks,
silently discarding them.
Fix contributed by Taylor Nelson.
- :mod:`celery.state`: Optimized precedence lookup.
Contributed by Matt Robenolt.
- Posix: Daemonization did not redirect ``sys.stdin`` to ``/dev/null``.
Fix contributed by Alexander Smirnov.
- Canvas: group bug caused fallback to default app when ``.apply_async`` used
(Issue #1516)
- Canvas: generator arguments was not always pickleable.
.. _version-3.0.22:
3.0.22
======
:release-date: 2013-08-16 16:30 P.M BST
- Now depends on :ref:`Kombu 2.5.13 <kombu:version-2.5.13>`.
- Now depends on :mod:`billiard` 2.7.3.32
- Fixed bug with monthly and yearly crontabs (Issue #1465).
Fix contributed by Guillaume Gauvrit.
- Fixed memory leak caused by time limits (Issue #1129, Issue #1427)
- Worker will now sleep if being restarted more than 5 times
in one second to avoid spamming with ``worker-online`` events.
- Includes documentation fixes
Contributed by: Ken Fromm, Andreas Savvides, Alex Kiriukha,
Michael Fladischer.
.. _version-3.0.21:
3.0.21
======
:release-date: 2013-07-05 16:30 P.M BST
- Now depends on :mod:`billiard` 2.7.3.31.
This version fixed a bug when running without the billiard C extension.
- 3.0.20 broke eventlet/gevent support (worker not starting).
- Fixed memory leak problem when MongoDB result backend was used with the
gevent pool.
Fix contributed by Ross Lawley.
.. _version-3.0.20:
3.0.20
======
:release-date: 2013-06-28 16:00 P.M BST
- Contains workaround for deadlock problems.
A better solution will be part of Celery 3.1.
- Now depends on :ref:`Kombu 2.5.12 <kombu:version-2.5.12>`.
- Now depends on :mod:`billiard` 2.7.3.30.
- ``--loader`` argument no longer supported importing loaders from the
current directory.
- [Worker] Fixed memory leak when restarting after connection lost
(Issue #1325).
- [Worker] Fixed UnicodeDecodeError at startup (Issue #1373).
Fix contributed by Jessica Tallon.
- [Worker] Now properly rewrites unpickleable exceptions again.
- Fixed possible race condition when evicting items from the revoked task set.
- [generic-init.d] Fixed compatibility with Ubuntu's minimal Dash
shell (Issue #1387).
Fix contributed by monkut.
- ``Task.apply``/``ALWAYS_EAGER`` now also executes callbacks and errbacks
(Issue #1336).
- [Worker] The :signal:`worker-shutdown` signal was no longer being dispatched
(Issue #1339)j
- [Python 3] Fixed problem with threading.Event.
Fix contributed by Xavier Ordoquy.
- [Python 3] Now handles ``io.UnsupportedOperation`` that may be raised
by ``file.fileno()`` in Python 3.
- [Python 3] Fixed problem with qualname.
- [events.State] Now ignores unknown event-groups.
- [MongoDB backend] No longer uses deprecated ``safe`` parameter.
Fix contributed by rfkrocktk
- The eventlet pool now imports on Windows.
- [Canvas] Fixed regression where immutable chord members may receive
arguments (Issue #1340).
Fix contributed by Peter Brook.
- [Canvas] chain now accepts generator argument again (Issue #1319).
- ``celery.migrate`` command now consumes from all queues if no queues
specified.
Fix contributed by John Watson.
.. _version-3.0.19:
3.0.19
======
:release-date: 2013-04-17 04:30:00 P.M BST
- Now depends on :mod:`billiard` 2.7.3.28
- A Python 3 related fix managed to disable the deadlock fix
announced in 3.0.18.
Tests have been added to make sure this does not happen again.
- Task retry policy: Default max_retries is now 3.
This ensures clients will not be hanging while the broker is down.
.. note::
You can set a longer retry for the worker by
using the :signal:`celeryd_after_setup` signal:
.. code-block:: python
from celery.signals import celeryd_after_setup
@celeryd_after_setup.connect
def configure_worker(instance, conf, **kwargs):
conf.CELERY_TASK_PUBLISH_RETRY_POLICY = {
'max_retries': 100,
'interval_start': 0,
'interval_max': 1,
'interval_step': 0.2,
}
- Worker: Will now properly display message body in error messages
even if the body is a buffer instance.
- 3.0.18 broke the MongoDB result backend (Issue #1303).
.. _version-3.0.18:
3.0.18
======
:release-date: 2013-04-12 05:00:00 P.M BST
- Now depends on :mod:`kombu` 2.5.10.
See the :ref:`kombu changelog <kombu:version-2.5.10>`.
- Now depends on :mod:`billiard` 2.7.3.27.
- Can now specify a whitelist of accepted serializers using
the new :setting:`CELERY_ACCEPT_CONTENT` setting.
This means that you can force the worker to discard messages
serialized with pickle and other untrusted serializers.
For example to only allow JSON serialized messages use::
CELERY_ACCEPT_CONTENT = ['json']
you can also specify MIME types in the whitelist::
CELERY_ACCEPT_CONTENT = ['application/json']
- Fixed deadlock in multiprocessing's pool caused by the
semaphore not being released when terminated by signal.
- Processes Pool: It's now possible to debug pool processes using GDB.
- ``celery report`` now censors possibly secret settings, like passwords
and secret tokens.
You should still check the output before pasting anything
on the internet.
- Connection URLs now ignore multiple '+' tokens.
- Worker/statedb: Now uses pickle protocol 2 (Py2.5+)
- Fixed Python 3 compatibility issues.
- Worker: A warning is now given if a worker is started with the
same node name as an existing worker.
- Worker: Fixed a deadlock that could occur while revoking tasks (Issue #1297).
- Worker: The :sig:`HUP` handler now closes all open file descriptors
before restarting to ensure file descriptors does not leak (Issue #1270).
- Worker: Optimized storing/loading the revoked tasks list (Issue #1289).
After this change the ``--statedb`` file will take up more disk space,
but loading from and storing the revoked tasks will be considerably
faster (what before took 5 minutes will now take less than a second).
- Celery will now suggest alternatives if there's a typo in the
broker transport name (e.g. ``ampq`` -> ``amqp``).
- Worker: The auto-reloader would cause a crash if a monitored file
was unlinked.
Fix contributed by Agris Ameriks.
- Fixed AsyncResult pickling error.
Fix contributed by Thomas Minor.
- Fixed handling of Unicode in logging output when using log colors
(Issue #427).
- :class:`~celery.app.utils.ConfigurationView` is now a ``MutableMapping``.
Contributed by Aaron Harnly.
- Fixed memory leak in LRU cache implementation.
Fix contributed by Romuald Brunet.
- ``celery.contrib.rdb``: Now works when sockets are in non-blocking mode.
Fix contributed by Theo Spears.
- The `inspect reserved` remote control command included active (started) tasks
with the reserved tasks (Issue #1030).
- The :signal:`task_failure` signal received a modified traceback object
meant for pickling purposes, this has been fixed so that it now
receives the real traceback instead.
- The ``@task`` decorator silently ignored positional arguments,
it now raises the expected :exc:`TypeError` instead (Issue #1125).
- The worker will now properly handle messages with invalid
eta/expires fields (Issue #1232).
- The ``pool_restart`` remote control command now reports
an error if the :setting:`CELERYD_POOL_RESTARTS` setting is not set.
- ``celery.conf.add_defaults`` can now be used with non-dict objects.
- Fixed compatibility problems in the Proxy class (Issue #1087).
The class attributes ``__module__``, ``__name__`` and ``__doc__``
are now meaningful string objects.
Thanks to Marius Gedminas.
- MongoDB Backend: The :setting:`MONGODB_BACKEND_SETTINGS` setting
now accepts a ``option`` key that lets you forward arbitrary kwargs
to the underlying ``pymongo.Connection` object (Issue #1015).
- Beat: The daily backend cleanup task is no longer enabled
for result backends that support automatic result expiration (Issue #1031).
- Canvas list operations now takes application instance from the first
task in the list, instead of depending on the ``current_app`` (Issue #1249).
- Worker: Message decoding error log message now includes traceback
information.
- Worker: The startup banner now includes system platform.
- ``celery inspect|status|control`` now gives an error if used
with an SQL based broker transport.
.. _version-3.0.17:
3.0.17
======
:release-date: 2013-03-22 04:00:00 P.M UTC
- Now depends on kombu 2.5.8
- Now depends on billiard 2.7.3.23
- RabbitMQ/Redis: thread-less and lock-free rate-limit implementation.
This means that rate limits pose minimal overhead when used with
RabbitMQ/Redis or future transports using the eventloop,
and that the rate-limit implementation is now thread-less and lock-free.
The thread-based transports will still use the old implementation for
now, but the plan is to use the timer also for other
broker transports in Celery 3.1.
- Rate limits now works with eventlet/gevent if using RabbitMQ/Redis as the
broker.
- A regression caused ``task.retry`` to ignore additional keyword arguments.
Extra keyword arguments are now used as execution options again.
Fix contributed by Simon Engledew.
- Windows: Fixed problem with the worker trying to pickle the Django settings
module at worker startup.
- generic-init.d: No longer double quotes ``$CELERYD_CHDIR`` (Issue #1235).
- generic-init.d: Removes bash-specific syntax.
Fix contributed by Pär Wieslander.
- Cassandra Result Backend: Now handles the
:exc:`~pycassa.AllServersUnavailable` error (Issue #1010).
Fix contributed by Jared Biel.
- Result: Now properly forwards apps to GroupResults when deserializing
(Issue #1249).
Fix contributed by Charles-Axel Dein.
- ``GroupResult.revoke`` now supports the ``terminate`` and ``signal``
keyword arguments.
- Worker: Multiprocessing pool workers now import task modules/configuration
before setting up the logging system so that logging signals can be
connected before they're dispatched.
- chord: The ``AsyncResult`` instance returned now has its ``parent``
attribute set to the header ``GroupResult``.
This is consistent with how ``chain`` works.
.. _version-3.0.16:
3.0.16
======
:release-date: 2013-03-07 04:00:00 P.M UTC
- Happy International Women's Day!
We have a long way to go, so this is a chance for you to get involved in one
of the organizations working for making our communities more
diverse.
- PyLadies — http://pyladies.com
- Girls Who Code — http://www.girlswhocode.com
- Women Who Code — http://www.meetup.com/Women-Who-Code-SF/
- Now depends on :mod:`kombu` version 2.5.7
- Now depends on :mod:`billiard` version 2.7.3.22
- AMQP heartbeats are now disabled by default.
Some users experiences issues with heartbeats enabled,
and it's not strictly necessary to use them.
If you're experiencing problems detecting connection failures,
you can re-enable heartbeats by configuring the :setting:`BROKER_HEARTBEAT`
setting.
- Worker: Now propagates connection errors occurring in multiprocessing
callbacks, so that the connection can be reset (Issue #1226).
- Worker: Now propagates connection errors occurring in timer callbacks,
so that the connection can be reset.
- The modules in :setting:`CELERY_IMPORTS` and :setting:`CELERY_INCLUDE`
are now imported in the original order (Issue #1161).
The modules in :setting:`CELERY_IMPORTS` will be imported first,
then continued by :setting:`CELERY_INCLUDE`.
Thanks to Joey Wilhelm.
- New bash completion for ``celery`` available in the git repository:
https://github.com/celery/celery/tree/3.0/extra/bash-completion
You can source this file or put it in ``bash_completion.d`` to
get auto-completion for the ``celery`` command-line utility.
- The node name of a worker can now include unicode characters (Issue #1186).
- The repr of a ``crontab`` object now displays correctly (Issue #972).
- ``events.State`` no longer modifies the original event dictionary.
- No longer uses ``Logger.warn`` deprecated in Python 3.
- Cache Backend: Now works with chords again (Issue #1094).
- Chord unlock now handles errors occurring while calling the callback.
- Generic worker init.d script: Status check is now performed by
querying the pid of the instance instead of sending messages.
Contributed by Milen Pavlov.
- Improved init scripts for CentOS.
- Updated to support celery 3.x conventions.
- Now uses CentOS built-in ``status`` and ``killproc``
- Support for multi-node / multi-pid worker services.
- Standard color-coded CentOS service-init output.
- A test suite.
Contributed by Milen Pavlov.
- ``ResultSet.join`` now always works with empty result set (Issue #1219).
- A ``group`` consisting of a single task is now supported (Issue #1219).
- Now supports the ``pycallgraph`` program (Issue #1051).
- Fixed Jython compatibility problems.
- Django tutorial: Now mentions that the example app must be added to
``INSTALLED_APPS`` (Issue #1192).
.. _version-3.0.15:
3.0.15
======
:release-date: 2013-02-11 04:30:00 P.M UTC
- Now depends on billiard 2.7.3.21 which fixed a syntax error crash.
- Fixed bug with :setting:`CELERY_SEND_TASK_SENT_EVENT`.
.. _version-3.0.14:
3.0.14
======
:release-date: 2013-02-08 05:00:00 P.M UTC
- Now depends on Kombu 2.5.6
- Now depends on billiard 2.7.3.20
- ``execv`` is now disabled by default.
It was causing too many problems for users, you can still enable
it using the :setting:`CELERYD_FORCE_EXECV` setting.
execv was only enabled when transports other than amqp/redis was used,
and it's there to prevent deadlocks caused by mutexes not being released
before the process forks. Sadly it also changes the environment
introducing many corner case bugs that is hard to fix without adding
horrible hacks. Deadlock issues are reported far less often than the
bugs that execv are causing, so we now disable it by default.
Work is in motion to create non-blocking versions of these transports
so that execv is not necessary (which is the situation with the amqp
and redis broker transports)
- Chord exception behavior defined (Issue #1172).
From Celery 3.1 the chord callback will change state to FAILURE
when a task part of a chord raises an exception.
It was never documented what happens in this case,
and the actual behavior was very unsatisfactory, indeed
it will just forward the exception value to the chord callback.
For backward compatibility reasons we do not change to the new
behavior in a bugfix release, even if the current behavior was
never documented. Instead you can enable the
:setting:`CELERY_CHORD_PROPAGATES` setting to get the new behavior
that will be default from Celery 3.1.
See more at :ref:`chord-errors`.
- worker: Fixes bug with ignored and retried tasks.
The ``on_chord_part_return`` and ``Task.after_return`` callbacks,
nor the ``task_postrun`` signal should be called when the task was
retried/ignored.
Fix contributed by Vlad.
- ``GroupResult.join_native`` now respects the ``propagate`` argument.
- ``subtask.id`` added as an alias to ``subtask['options'].id``
.. code-block:: python
>>> s = add.s(2, 2)
>>> s.id = 'my-id'
>>> s['options']
{'task_id': 'my-id'}
>>> s.id
'my-id'
- worker: Fixed error `Could not start worker processes` occurring
when restarting after connection failure (Issue #1118).
- Adds new signal :signal:`task-retried` (Issue #1169).
- `celery events --dumper` now handles connection loss.
- Will now retry sending the task-sent event in case of connection failure.
- amqp backend: Now uses ``Message.requeue`` instead of republishing
the message after poll.
- New :setting:`BROKER_HEARTBEAT_CHECKRATE` setting introduced to modify the
rate at which broker connection heartbeats are monitored.
The default value was also changed from 3.0 to 2.0.
- :class:`celery.events.state.State` is now pickleable.
Fix contributed by Mher Movsisyan.
- :class:`celery.datastructures.LRUCache` is now pickleable.
Fix contributed by Mher Movsisyan.
- The stats broadcast command now includes the workers pid.
Contributed by Mher Movsisyan.
- New ``conf`` remote control command to get a workers current configuration.
Contributed by Mher Movsisyan.
- Adds the ability to modify the chord unlock task's countdown
argument (Issue #1146).
Contributed by Jun Sakai
- beat: The scheduler now uses the `now()`` method of the schedule,
so that schedules can provide a custom way to get the current date and time.
Contributed by Raphaël Slinckx
- Fixed pickling of configuration modules on Windows or when execv is used
(Issue #1126).
- Multiprocessing logger is now configured with loglevel ``ERROR``
by default.
Since 3.0 the multiprocessing loggers were disabled by default
(only configured when the :envvar:`MP_LOG` environment variable was set).
.. _version-3.0.13:
3.0.13
======
:release-date: 2013-01-07 04:00:00 P.M UTC
- Now depends on Kombu 2.5
- py-amqp has replaced amqplib as the default transport,
gaining support for AMQP 0.9, and the RabbitMQ extensions
including Consumer Cancel Notifications and heartbeats.
- support for multiple connection URLs for failover.
- Read more in the :ref:`Kombu 2.5 changelog <kombu:version-2.5.0>`.
- Now depends on billiard 2.7.3.19
- Fixed a deadlock issue that could occur when the producer pool
inherited the connection pool instance of the parent process.
- The :option:`--loader` option now works again (Issue #1066).
- :program:`celery` umbrella command: All subcommands now supports
the :option:`--workdir` option (Issue #1063).
- Groups included in chains now give GroupResults (Issue #1057)
Previously it would incorrectly add a regular result instead of a group
result, but now this works:
.. code-block:: python
# [4 + 4, 4 + 8, 16 + 8]
>>> res = (add.s(2, 2) | group(add.s(4), add.s(8), add.s(16)))()
>>> res
<GroupResult: a0acf905-c704-499e-b03a-8d445e6398f7 [
4346501c-cb99-4ad8-8577-12256c7a22b1,
b12ead10-a622-4d44-86e9-3193a778f345,
26c7a420-11f3-4b33-8fac-66cd3b62abfd]>
- Chains can now chain other chains and use partial arguments (Issue #1057).
Example:
.. code-block:: python
>>> c1 = (add.s(2) | add.s(4))
>>> c2 = (add.s(8) | add.s(16))
>>> c3 = (c1 | c2)
# 8 + 2 + 4 + 8 + 16
>>> assert c3(8).get() == 38
- Subtasks can now be used with unregistered tasks.
You can specify subtasks even if you just have the name::
>>> s = subtask(task_name, args=(), kwargs=())
>>> s.delay()
- The :program:`celery shell` command now always adds the current
directory to the module path.
- The worker will now properly handle the :exc:`pytz.AmbiguousTimeError`
exception raised when an ETA/countdown is prepared while being in DST
transition (Issue #1061).
- force_execv: Now makes sure that task symbols in the original
task modules will always use the correct app instance (Issue #1072).
- AMQP Backend: Now republishes result messages that have been polled
(using ``result.ready()`` and friends, ``result.get()`` will not do this
in this version).
- Crontab schedule values can now "wrap around"
This means that values like ``11-1`` translates to ``[11, 12, 1]``.
Contributed by Loren Abrams.
- multi stopwait command now shows the pid of processes.
Contributed by Loren Abrams.
- Handling of ETA/countdown fixed when the :setting:`CELERY_ENABLE_UTC`
setting is disabled (Issue #1065).
- A number of uneeded properties were included in messages,
caused by accidentally passing ``Queue.as_dict`` as message properties.
- Rate limit values can now be float
This also extends the string format so that values like ``"0.5/s"`` works.
Contributed by Christoph Krybus
- Fixed a typo in the broadcast routing documentation (Issue #1026).
- Rewrote confusing section about idempotence in the task user guide.
- Fixed typo in the daemonization tutorial (Issue #1055).
- Fixed several typos in the documentation.
Contributed by Marius Gedminas.
- Batches: Now works when using the eventlet pool.
Fix contributed by Thomas Grainger.
- Batches: Added example sending results to :mod:`celery.contrib.batches`.
Contributed by Thomas Grainger.
- Mongodb backend: Connection ``max_pool_size`` can now be set in
:setting:`CELERY_MONGODB_BACKEND_SETTINGS`.
Contributed by Craig Younkins.
- Fixed problem when using earlier versions of :mod:`pytz`.
Fix contributed by Vlad.
- Docs updated to include the default value for the
:setting:`CELERY_TASK_RESULT_EXPIRES` setting.
- Improvements to the django-celery tutorial.
Contributed by Locker537.
- The ``add_consumer`` control command did not properly persist
the addition of new queues so that they survived connection failure
(Issue #1079).
3.0.12
======
:release-date: 2012-11-06 02:00 P.M UTC
- Now depends on kombu 2.4.8
- [Redis] New and improved fair queue cycle algorithm (Kevin McCarthy).
- [Redis] Now uses a Redis-based mutex when restoring messages.
- [Redis] Number of messages that can be restored in one interval is no
longer limited (but can be set using the
``unacked_restore_limit``
:setting:`transport option <BROKER_TRANSPORT_OPTIONS>`.)
- Heartbeat value can be specified in broker URLs (Mher Movsisyan).
- Fixed problem with msgpack on Python 3 (Jasper Bryant-Greene).
- Now depends on billiard 2.7.3.18
- Celery can now be used with static analysis tools like PyDev/PyCharm/pylint
etc.
- Development documentation has moved to Read The Docs.
The new URL is: http://docs.celeryproject.org/en/master
- New :setting:`CELERY_QUEUE_HA_POLICY` setting used to set the default
HA policy for queues when using RabbitMQ.
- New method ``Task.subtask_from_request`` returns a subtask using the current
request.
- Results get_many method did not respect timeout argument.
Fix contributed by Remigiusz Modrzejewski
- generic_init.d scripts now support setting :envvar:`CELERY_CREATE_DIRS` to
always create log and pid directories (Issue #1045).
This can be set in your :file:`/etc/default/celeryd`.
- Fixed strange kombu import problem on Python 3.2 (Issue #1034).
- Worker: ETA scheduler now uses millisecond precision (Issue #1040).
- The ``--config`` argument to programs is now supported by all loaders.
- The :setting:`CASSANDRA_OPTIONS` setting has now been documented.
Contributed by Jared Biel.
- Task methods (:mod:`celery.contrib.methods`) cannot be used with the old
task base class, the task decorator in that module now inherits from the new.
- An optimization was too eager and caused some logging messages to never emit.
- :mod:`celery.contrib.batches` now works again.
- Fixed missing whitespace in ``bdist_rpm`` requirements (Issue #1046).
- Event state's ``tasks_by_name`` applied limit before filtering by name.
Fix contributed by Alexander A. Sosnovskiy.
.. _version-3.0.11:
3.0.11
======
:release-date: 2012-09-26 04:00 P.M UTC
- [security:low] generic-init.d scripts changed permissions of /var/log & /var/run
In the daemonization tutorial the recommended directories were as follows:
.. code-block:: bash
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
But in the scripts themselves the default files were ``/var/log/celery%n.log``
and ``/var/run/celery%n.pid``, so if the user did not change the location
by configuration, the directories ``/var/log`` and ``/var/run`` would be
created - and worse have their permissions and owners changed.
This change means that:
- Default pid file is ``/var/run/celery/%n.pid``
- Default log file is ``/var/log/celery/%n.log``
- The directories are only created and have their permissions
changed if *no custom locations are set*.
Users can force paths to be created by calling the ``create-paths``
subcommand:
.. code-block:: bash
$ sudo /etc/init.d/celeryd create-paths
.. admonition:: Upgrading Celery will not update init scripts
To update the init scripts you have to re-download
the files from source control and update them manually.
You can find the init scripts for version 3.0.x at:
http://github.com/celery/celery/tree/3.0/extra/generic-init.d
- Now depends on billiard 2.7.3.17
- Fixes request stack protection when app is initialized more than
once (Issue #1003).
- ETA tasks now properly works when system timezone is not the same
as the configured timezone (Issue #1004).
- Terminating a task now works if the task has been sent to the
pool but not yet acknowledged by a pool process (Issue #1007).
Fix contributed by Alexey Zatelepin
- Terminating a task now properly updates the state of the task to revoked,
and sends a ``task-revoked`` event.
- Generic worker init script now waits for workers to shutdown by default.
- Multi: No longer parses --app option (Issue #1008).
- Multi: stop_verify command renamed to stopwait.
- Daemonization: Now delays trying to create pidfile/logfile until after
the working directory has been changed into.
- :program:`celery worker` and :program:`celery beat` commands now respects
the :option:`--no-color` option (Issue #999).
- Fixed typos in eventlet examples (Issue #1000)
Fix contributed by Bryan Bishop.
Congratulations on opening bug #1000!
- Tasks that raise :exc:`~celery.exceptions.Ignore` are now acknowledged.
- Beat: Now shows the name of the entry in ``sending due task`` logs.
.. _version-3.0.10:
3.0.10
======
:release-date: 2012-09-20 05:30 P.M BST
- Now depends on kombu 2.4.7
- Now depends on billiard 2.7.3.14
- Fixes crash at startup when using Django and pre-1.4 projects
(setup_environ).
- Hard time limits now sends the KILL signal shortly after TERM,
to terminate processes that have signal handlers blocked by C extensions.
- Billiard now installs even if the C extension cannot be built.
It's still recommended to build the C extension if you are using
a transport other than rabbitmq/redis (or use forced execv for some
other reason).
- Pool now sets a ``current_process().index`` attribute that can be used to create
as many log files as there are processes in the pool.
- Canvas: chord/group/chain no longer modifies the state when called
Previously calling a chord/group/chain would modify the ids of subtasks
so that:
.. code-block:: python
>>> c = chord([add.s(2, 2), add.s(4, 4)], xsum.s())
>>> c()
>>> c() <-- call again
at the second time the ids for the tasks would be the same as in the
previous invocation. This is now fixed, so that calling a subtask
won't mutate any options.
- Canvas: Chaining a chord to another task now works (Issue #965).
- Worker: Fixed a bug where the request stack could be corrupted if
relative imports are used.
Problem usually manifested itself as an exception while trying to