Server was slow this morning, server upgrade project?

Hi all,

When I woke up this morning, I realized that the site was slow.

I found the issue and i'm curently patching the problem like we did the
last time.

All should be back to normal.

While we are at it, I am outlining below an upgrade project, and i'd
like your input on it.



*The problem:*

It is caused by Semantic Media Wiki ("SMW") that is spawning a never
ending set of tasks.

When somebody updates a template, SMW creates jobs in a queue to update
the views. A cronjob picks them up and processes them. Generally it
takes a minute and has no more than a hundred. 

But we've been experiencing this for a while now. When we update some
popular templates, the original jobs are created like usual. But the
issue we think SMW has is that we think that when a job is finished,
some other jobs might be created. But in this particular case, we think
that that loop never ends.

In consequence, we end up with thousands of jobs and when we take too
much time to notice, we can see it piling up to millions!

It's not the first time that we are experiencing this. We know the issue
and we should upgrade our MediaWiki installation.


*The solution (long term)*

I have this project to upgrade the MW+SMW upgrade on hold. 

The current server setup for MediaWiki ("MW") is Ubuntu 10.04 LTS with
two MySQL 5.1 servers. One master, one read-only.

I'd like that next upgrade, we upgrade more than one layer. We could
upgrade to latest MySQL 5.6, Ubuntu 14.04 LTS that got out a week ago
and upgrade our MW+SMW distribution.

At this time, I have a sandbox where I was practicing the local
development stack [2][3], and testing in the same time on a sandbox VM
behind the caching layer. You can see the sandbox at
http://sandbox.webplatform.org/test/

Please don't mind the error messages, I keep them on to see what needs
adjusting.

You will see that it runs a different version of MediaWiki [0] from the
live site [1].

I can say that the  Wikipedia people are currently using similar
environment that i'm suggesting. They are even upgrading MediaWiki
almost more than once a week.

Maybe I should prioritize it soon enough. 

I'm not saying we should upgrade every week, but I'd like to see with
you which components we can remove to lighten our setup and ease the
upgrades.

What do you think?


  [0]: http://sandbox.webplatform.org/test/Special:Version
  [1]: http://docs.webplatform.org/wiki/Special:Version
  [2]: https://github.com/webplatform/mediawiki
  [3]: https://github.com/webplatform/saltyvagrant-mediawiki

-- 
Regards,

Renoir Boulanger  |  Developer operations engineer
W3C  |  Web Platform Project

http://w3.org/people/#renoirb  ✪  https://renoirboulanger.com/  ✪  @renoirb
~