Re: Attempt to solve the job runner situation

On Wed, Nov 6, 2013 at 12:30 AM, Renoir Boulanger <renoir@w3.org> wrote:

> Hi all,
>
> Tonight, between 23:00 to 24:30 EST, Ryan and I had the chance to work on
> the job runner situation [0].
>
> The problem was two-fold. Part of the problem was due to a high level of
> database writes that was filling the database server storage partitions.
> The main reason of the problem was because of an ever increasing quantity
> of jobs being added to MediaWiki's job runner system. Details of the issue
> are described in [0].
>
> Symptoms of the problem were described in those e-mails: [1][2][3]
>
> Ryan took the decision to truncate SemanticMediaWiki data and delete all
> SemanticMediaWiki related jobs.
>
> We hope it solves the situation.
>
>
The problem is indeed solved. I believe the SMW data was corrupted, causing
it to continuously add jobs into the queue to fix its data. I truncated the
job table, then refreshed the SMW data using a maintenance script. The
current job queue is at 0.

- Ryan