FusionForge

Reactivity

From FusionForge Wiki
Revision as of 09:23, 22 September 2015 by Beuc-inria (talk | contribs) (tag)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Initial discussion: Meetings/2014#Reactivity

Question is raised on the benefits of using an existing queue system vs. using a simple SQL poller:

  • Multi-servers distribution
  • Multi-actions for a single message

Idea:

  • use a simple 1 task -> 1 action on 1 server -> ACK visible by the user through the WebUI (easy to implement and to check the status of a task)
  • special-case nscd reload (mode publish-subscribe)
  • restart the daemon on upgrades

Bad ideas:

  • publish-subscribe everywhere: means that actions are not done if the queue daemon (or the server) is down for e.g. maintenance
  • 1 task -> N actions (on possibly N servers): makes it difficult to check if a task is completed, this would requires pre-configuring a centralised list of servers impacted by each task
  • store the script name in the database: to prevent a DB injection from escalating to a "chmod u+s", the script names will be hardcoded in the source code, and each will have a unique ID.

Limitations:

  • On a multi-server setup, the homedirs (users and groups) will be created once. The system won't create the homedirs on each servers, so sou need to use a network share (e.g. NFS).

Possible improvements:

  • Instead of managing task IDs (e.g. 27), directly use directly the variable identifier (e.g. SYSACTION_SCM_REPO) in the DB
  • Rename sysaction -> systask?
  • Only activate 1 wiki in mediawiki/cronjobs/create-wikis.php (rather than recheck all wikis)
    • merge create-wikis.php and create-imagedirs.php


Message system:

  • Store message in the queue
  • Perform the tasks one by one vs. launch a worker in the background
  • ACK on completion
  •  ?What to do when the task fails?
  • Provide feedback to the end-user (view on the queue) -> makes it difficult to integrate with an existing queue daemon such as Rabbit MQ.


Data structure:

  • Task:
    • task_id / script name ??
    • status: TODO / WIP / DONE / ERROR
    • request_timestamp
    • error_message
    • user_id/group_id? (to give feedback to a user about the tasks that impact him/her)
  • Special case: nscd reload
    • last_modified (queue deamon restarts nscd 1) on startup and 2) if last_modified changes)
  • Additional tables specific to each task type:
    • mail_group_list
    • scm_secondary_repos
    • sysactionsq_xxx...

Crons to convert:

  • Add a user
    • user homedir
  • Add/remove a group
    • group homedir
    • invalidating nscd cache, plus restart Apache for current InriaForge (to be replaced by itk)
      • SCM browser
      • Access over HTTP(s)
      • Access over SSH
  • Add/remove a member
    • invalidating nscd cache, plus restart Apache (updated itk conf)
  • Deactivate/reactive a user or group
    • invalidating nscd cache, plus restart Apache (updated itk conf)
  • [X] SSH replication [AuthorizedKeysCommand]
  • [X] SCM repository creation
    • trigger SCM task when role anonymous is added or removed from a group
    • actually whenever there's a change in the roles, since we have to regenerate (not reload) the apache config and password files
    • [X] optimise building SVN access file; it's way too long to be run each time a project membership is modified -> 'svnroot-access' ditched as we can rely on user perms with ITK
  • [X] vhosts
  • [X] Mailing lists creation/deletion/password-reset
  • [X] Mass-mailing
  • [X] Plugin Mediawiki (wiki creation) (not nightly dumps)
  • [X] Plugin SCMHook
  • [-] Plugin MoinMoin (wiki creation)

Keep in cron:

  • db: stats (aggregations, mail reports), tracker items closing...
  • SCM and wiki daily snapshots / tarballs
  • SCM processes clean-up (scmsvn)