=head1  NAME

Cpanel::TaskQueue - FIFO queue of tasks to perform

=head1 VERSION

This document describes Cpanel::TaskQueue version 0.307

=head1 SYNOPSIS

    use Cpanel::TaskQueue ();

    my $queue = Cpanel::TaskQueue->new( { name => 'tasks', cache_dir => "/home/$user/.cpanel/cache" } );

    $queue->queue_task( "init_quota" );
    $queue->queue_task( "edit_quota fred 0" );

    # Processing loop
    while (1) {
        # if work, process, else sleep
        if ( $queue-has_work_to_do() ) {
            eval { $queue->process_next_task() };
            if ( $@ ) {
                Carp::carp( $@ );
            }
        }
        else {
            # wait for work.
            sleep 300;
        }
    }

=head1  DESCRIPTION

This module provides an abstraction for a FIFO queue of tasks that may be
executed asynchronously. Each command determines whether it runs in the
current process or forks a child to do the work in the background.

The TaskQueue has support for limiting the number of background tasks running
at one time and for preventing duplicate tasks from being scheduled.

=head1 PUBLIC METHODS

=over 4

=item Cpanel::TaskQueue->new( $hashref )

Creates a new TaskQueue object based on the parameters from the supplied hashref.

=over 4

=item I<cache_dir>

This required parameter specifies a directory where the cache should be written.
This directory is created if it does not exist.

=item I<name>

This required parameter specifies the name of the queue. This name is used to
construct the name of the cache file used to store the queue information.

=item I<cache_timeout>

This optional parameter specifies the timeout to use for flocking the cache file.
The value is in seconds and defaults to the Cpanel::CacheFile default value.

=item I<default_timeout>

This optional parameter specifies the default amount of time (in seconds) to wait
for an in-process task to run. The default value is 60 seconds.
    
This timeout may be overridden for a particular command by the
C<Cpanel::TaskQueue::Processor> object.

=item I<max_timeout>

This optional parameter specifies the maximum amount of time (in seconds) to wait
for an in-process task to run. The default value is 300 seconds (5 minutes).

This value prevents a given C<Cpanel::TaskQueue::Processor> object from setting the timeout
too high.

=item I<max_running>

This optional parameter specifies the maximum number of child processes that the
queue will allow to be running at one time. If this number is reached, all queue
processing blocks until one or more child processes exit. The default value is 2.

=item I<default_child_timeout>

This optional parameter specifies the default amount of time (in seconds) to wait
for a child task to run. The default value is 3600 seconds (1 hour).
    
This timeout may be overridden for a particular command by the
C<Cpanel::TaskQueue::Processor> object.

=back

If these parameters are not specified, but a C<TaskQueue> with this I<name> and
I<cache_dir> has already been created, the new C<TaskQueue> will use the parameters
that were stored in the file.  This causes all instances of this C<TaskQueue>
to act the same. Providing these parameters also updates all other instances of
this C<TaskQueue> the next time they need to C<synch>.

=item $q->queue_task( $command )

Create a new task from the command and put it at the end of the queue if it meets
certain minimum criteria.

=over 4

=item Command must be legal.

The command type must have been registered with the TaskQueue module.

=item Command must not duplicate a command already in the queue.

Each command type can have it's own definition of duplicate. It can depend on
one or more of the arguments or not.

=back

If the task is successfully queued, a non-empty I<uuid> is returned. This id can
be used to remove the task from the queue at a later time.

If the task was not queued, a false value is returned.

The C<queue_task> method can also be called with a C<Cpanel::TaskQueue::Task>
object which will be tested and inserted as usual.

=item $q->unqueue_task( $uuid )

Remove the task associated with the supplied I<uuid> from the queue, if it
has not been processed yet. Returns true on success.

=back

=head2 QUEUE PROCESSING

=over 4

=item $q->has_work_to_do()

Returns a true value if there are any tasks in the queue and we can process them.
This method does not block on child processes that are currently running if we
cannot launch a new task.

=item $q->process_next_task()

This method is called to process another task from the wait queue.

If there are any tasks remaining and we have not reached the limit of tasks we
can process at once, the next is removed from the queue. The task is checked to
make certain we know how to process it, if not it is discarded.  Then it is added
to the processing list and the C<Cpanel::TaskQueue::Processor> object for that
command is asked to process it. If we have reached our processing limit, block
until a task can be executed.

If the command is completed by the C<Cpanel::TaskQueue::Processor>, the task is
removed from the processing list. If the C<Cpanel::TaskQueue::Processor> launched
a child process, the task is left in the processing list.

The method returns true if the task was completed or otherwise removed from the
system. If the task was launched as a child process, the method returns false. The
method will also return true if there is nothing to process.

=item $q->finish_all_processing()

This method does not return until all tasks currently being processed in the
background are completed. It is most useful to call as part of shutdown of the
program that processes the queue. While waiting for processing to complete,
this method does not start any new tasks out of the queue.

=back

=head2 QUEUE INFORMATION

=over 4

=item $q->get_default_child_timeout

Returns the default timeout value for a child process.

=item $q->get_default_timeout

Returns the default timeout value for a task.

=item $q->get_max_running

Returns the maximum number of child processes that can be running at once.

=item $q->get_max_timeout

Returns the maximum timeout value for a task.

=item $q->get_name

Returns the TaskQueue's name.

=item $q->peek_next_task()

Get a copy of the first task descriptor in the queue or C<undef> if the queue is
empty.

Because of the nature of a task queue, there is no guarantee that this task will
remain unscheduled after the method call. That is one reason that a copy is
returned.

=item $q->is_task_queued( $uuid )

Does the specified I<uuid> reference a task in the queue?

Because of the nature of a task queue, the particular I<uuid> tested may be scheduled
for processsing immediately after the test. Therefore, a true answer is not as useful as
it might seem. A false answer does tell us that the item is no longer waiting. 

=item $q->find_task( $uuid )

Returns a copy of the task in the queue with the supplied I<uuid>. Returns
C<undef> if no task with that I<uuid> is found. Because of the nature of the
task queue, the task that is returned may not be in the queue shortly after
return from this method. Another process may have handled it and removed it from
the queue.

However, the returned copy is a faithful representation of the task at the point
in time that it was found.

=item $q->find_command( $command )

Returns a copy of the first command with the supplied I<command> (sans
arguments).  Returns C<undef> if no task with that command name is found.
Because of the nature of the task queue, the task that is returned may not be
in the queue shortly after return from this method. Another process may have
handled it and removed it from the queue.

Remember that C<$command> is just the name of the command, not the whole
command string with arguments.

=item $q->how_many_queued()

Gives a count at this particular point in time of the number of items currently
in the queue. Since an item may be removed and processed any time the
C<process_next_task()> method is called, this count may not be correct immediately
after the method returns.

Most useful for the general case of telling if the queue is really full, or mostly
empty.

=item $q->is_task_processing( $uuid )

Does the specified I<uuid> reference a task currently being processed?

Because of the nature of a task queue, the particular I<uuid> tested may be scheduled
for processsing or finish processing immediately after the test. I'm not sure if this
test is actually useful for anything.

=item $q->how_many_in_process()

Returns a count of the number of items currently being processed. Since a task
can complete at any time, the exact value returned by this method is not
guaranteed for any length of time after the method returns. May be useful to get
a statistical measure of how busy the C<Cpanel::TaskQueue> system is.

=item $q->snapshot_task_lists()

Returns a reference to a hash containing copies of the current queues. The value
of I<waiting> is an array reference containing copies of all of the C<Task>s
waiting to execute. The value of I<processing> is an array reference containing
copies of all of the C<Tasks> currently being processed.

Since a task can complete at any time and whatever process handles the queue can
start processing a task at any time, the output of this method may be out of
date as soon as it returns. This method is only really useful for a general idea
of the state of the queue.

=back

=head2 CACHE SUPPORT

These methods should not be used directly, they exist to support the C<Cpanel::CacheFile>
interface that persists the queue information to disk.

=over 4

=item $q->load_from_cache( $fh )

This method loads the queue information from the disk cache. It is called by the
C<Cpanel::CacheFile> object owned by this object.

The user of this class should never need to call this method.

=item $q->save_to_cache( $fh )

This method saves the queue information to the disk cache. It is called by the
C<Cpanel::CacheFile> object owned by this object.

The user of this class should never need to call this method.

=item $q->throw( $msg )

Log the supplied message and C<die>.

=item $q->warn( $msg )

Log the supplied message as a warning.

=item $q->info( $msg )

Log the supplied message as an informational message.

=back

=head1 CLASS METHODS

The class also supports a few methods that apply to the Task Queuing system as a whole.
These methods manage the registering of task processing objects.

=over 4

=item Cpanel::TaskQueue->register_task_processor( $cmdname, $processor )

Add a task processing object for the command name given as the first argument.
The second argument must either be a C<Cpanel::TaskQueue::Processor>-derived object
or a code reference that will be wrapped in a C<Cpanel::TaskQueue::Processor::CodeRef>
object.

=item Cpanel::TaskQueue->unregister_task_processor( $cmdname )

Removes the task processing object for the command given as the only argument.

After a call to this method, that particular command can not be queued any more
and any already queued objects will be discarded when the C<Cpanel::TaskQueue>
tries to process them.

=back

=head1 LOGGER OBJECT

By default, the C<TaskQueue> uses C<die> and C<warn> for all messages during
runtime. However, it supports a mechanism that allows you to insert a
logging/reporting system in place by providing an object to do the logging for
us.

To provide a different method of logging/reporting, supply an object to do the
logging as follows when C<use>ing the module.

   use Cpanel::TaskQueue ( '-logger' => $logger );

The supplied object should supply (at least) 3 methods: C<throw>, C<warn>, and
C<info>. When needed these methods will be called with the messages to be logged.

The C<throw> method is expected to use C<die> to exit the method. The others
are expected to continue. For example, an appropriate class for C<Log::Log4perl>
might do something like the following:

    package Policy::Log4perl;
    use strict;
    use warnings;
    use Log::Log4perl;

    sub new {
        my ($class) = shift;
        my $self = {
            logger => Log::Log4perl->get_logger( @_ )
        };
        return bless, $class;
    }

    sub throw {
        my $self = shift;
        $self->{logger}->error( @_ );
        die @_;
    }

    sub warn {
        my $self = shift;
        $self->{logger}->warn( @_ );
    }

    sub info {
        my $self = shift;
        $self->{logger}->info( @_ );
    }

This would call the C<Log4perl> code as errors or other messages result in
messages.

This only works once for a given program, so you can't reset the policy in
multiple modules and expect it to work.

In addition to setting a global logger, a new logger object can be supplied
when creating a specific C<TaskQueue> object.

=head1 DIAGNOSTICS

The following messages can be reported by this module:

=over 4

=item C<< Missing command in register_task_processor. >>

No command name was given when calling the C<register_task_processor> class
method to register a processing object to handle a command.

=item C<< Missing task processor in register_task_processor. >>

No command processor object was supplied when calling the C<register_task_processor>
class method to register an action to attach to a command.

=item C<< Command '%s' already has a TaskQueue::Processor registered. >>

The supplied command name already has a registered processing object. If you want
to change it, you must first remove the other processor using
C<unregister_task_processor>.

=item C<< Unrecognized task processor object. >>

The second parameter to C<register_task_processor> was not recognized as a
C<TaskQueue::Processor>-derived object or as a C<coderef>.

=item C<< Missing command in unregister_task_processor. >>

No command name string was supplied when calling this method.

=item C<< Command '%s' not registered, ignoring. >>

The supplied argument to C<unregister_task_processor> was not a registered
command name.

=item C<< No caching directory supplied. >>

The required I<cache_dir> parameter was missing when constructing the C<TaskQueue>
object. The object was not created.

=item C<< No queue name supplied. >>

The required I<name> parameter was missing when constructing the C<TaskQueue>
object. The object was not created.

=item C<< Not a recognized TaskQueue cache. >>

=item C<< Invalid version of TaskQueue cache. >>

Either the cache file is invalid or it is not a TaskQueue cache file.

=item C<< Cannot queue an empty command. >>

The command string supplied to C<queue_task> was either C<undef> or empty.

=item C<< Task with 0 retries not queued. >>

The C<Task> supplied to C<queue_task> has a remaining retry count of 0. The task
has been discarded. This is a warning message only.

=item C<< No known processor for '%s'. >>

The specified command has no defined processor. The command has been discarded.

=item C<< Requested command [%s] has invalid arguments. >>

The supplied full command has arguments that are not valid as defined by the
command processor.

=item C<< No Task uuid argument passed to %s. >>

The specified method requires a I<uuid> to specify which task to operate
on. None was supplied.

=item C<< No processor found for '%s'. >>

Either the program inserting tasks into the queue has a different list of commands
than the program processing the queue, or a TaskQueue::Processor was unregistered
after this command was queued.

=item C<< Task '%s' timed out during processing. >>

The supplied command timed out while being executed in-process.

=item C<< Undefined tasks found in the queue, removing... >>

Somehow a task item of C<undef> has appeared in the queue. This should never
happen, so if it does, we remove them.

=back

=head1 DEPENDENCIES

YAML::Syck, POSIX

Cpanel::TaskQueue::Processor, Cpanel::TaskQueue::Task, Cpanel::CacheFile

=head1 INCOMPATIBILITIES

None reported.

=head1 BUGS AND LIMITATIONS

none reported.

=head1 SEE ALSO

Cpanel::TaskQueue::Processor, Cpanel::TaskQueue::Task, and Cpanel::CacheFile

=head1 LICENCE AND COPYRIGHT

Copyright (c) 2009, CPanel. All rights reserved.

This module is free software; you can redistribute it and/or
modify it under the same terms as Perl itself. See L<perlartistic>.

=head1 DISCLAIMER OF WARRANTY

BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE
ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH
YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL
NECESSARY SERVICING, REPAIR, OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE
LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL,
OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE
THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.