Servers, Servers Everywhere…

One of the cool things about Orbital from my point of view is that I’m not just responsible for putting together a bit of software that runs on a web server, but also for designing the reference platform which you run those bits of software on.

At this point I could digress into discussing exactly what boxes we’re running Orbital on top of, but that doesn’t really matter. What is more interesting is how the various servers click together into building the complete Orbital platform, and how those servers can help us scale and provide a resilient service.

You’re probably used to thinking of most web applications like this:

A 'traditional' server model

It’s simple. You install what you need to run your application on a server, hook it up to the internet, and off you go. Everything is contained on a single box which gives you epic simplicity benefits and is often a lot more cost efficient, but you lose scalability. If one day your application has a traffic spike your Serv-O-Matic 100 may not be able to cope. The solution is to make your server bigger!

Throw more power at it!

This is all well and good, until you start to factor in resiliency as well. Your Serv-O-Matic 500 may be sporting 16 processor cores and 96GB of RAM, but it’s only doing it’s job until the OS decides it’s going to fall over, or your network card gives up, or somebody knocks the Big Red Switch.

Continue reading “Servers, Servers Everywhere…”

Orbital is go! (Team meeting notes 24-11-2011)

Attending: Nick Jackson, Annalisa Jones, Bev Jones, Chris Leach, Paul Stainthorp, Joss Winn

Apologies: Lee Mitchell, David Young

Agenda

  1. Review Project Plan and Workpackages
  2. Status updates: Literature Review, User Requirements Analysis, Technology/Standards evaluation
  3. Forthcoming meetings and conferences (Agile method, Open Source policy, ERIM, Engineers, OR12, DCC, Start-up)
  4. Poster, papers, website
  5. Staffing and accommodation
  6. AOB

Notes

Joss Winn (JW) reported in detail:

  • JW reported on the work done to date (mostly relating to workpackage WP1), and reported back on:
    • The successful first meeting with users from the School of Engineering
    • The first Steering Group meeting on 3 November
    • The submission of the project plan
    • The appointment of NJ as lead developer
    • The relocation of NJ and PS (part-time) to CERD’s offices to work on Orbital
  • JW ran through the project outputs and workpackages in detail, identifying deadlines – most notably the Implementation Plan, which must be submitted by February 2012, with the following four pieces of work completed by then:
    • Data sources (NJ/CL)
    • User requirements (NJ)
    • Literature review (PS/BJ/CL)
    • Technical review (NJ/JW)
  • The group discussed the further user-engagement work to be completed in workpackages WP5, including Nick Jackson’s work with the School of Engineering to assess their requirements (through workshops, questionnaires, observation, and use of the Data Asset Framework – DAF), and on a planned round-table meeting about ERIM in late January
  • ACTION (NJ): dates needs to be set for user requirements exercises.
  • ACTION (PS): Date in late January needs to be set for ERIM workshop with Engineers.
  • PS reported on the work that he and NJ have begun to benchmark against the EPrints deposit workflow (WP8). NJ will work closely with BJ on this.
  • The group discussed WP9—the planned assessment of data sources—and CL’s role as library user. There are three obvious areas where Orbital crosses over with the Library’s priorities:
    • Integration with the Library’s Discovery selection & implementation project (CL)
    • Integration with the Repository (BJ)
    • Authentication (CL)
  • The Research & Enterprise office (i.e. AJ) will lead on WP11 – developing training materials & workshops.
  • JW will carry on the work with the University’s IP manager, James Murray on the correct approach to Open Sourcing code from Orbital – WP13.
  • ACTION: JW to follow up contacts with EPrints Services and OSSWatch.
  • Dissemination (WP14):
    • PS has been invited to speak at two events in January/February. The group will aim to have a publishable conference paper ready by Summer 2012. Submit abstract to OR12 by ?.
    • NJ, PS and JW are attending the project startup meeting in Nottingham on 1-2 December; presenting a poster. Also attending the DCC roadshow in Cardiff in mid-December.
  • Any other business:
    • JW is convening a meeting (8th December) about agile software and project development methods.
  • ACTION: as many people as possible from Orbital to attend ‘agile’ meeting.

Project Planning: Quality Assurance

I am currently completing the Orbital Project Plan prior to submission to JISC next week. The writing of a Project Plan, using JISC’s template, which I think is a derivation of PRINCE2 documentation, is undeniably a useful exercise in defining the project we’re embarking on. It is also undeniably a tedious process, too, as it requires a particular style of thinking and writing: granular, incremental and forward looking, yet reflective; ambitious and creative yet restrained; serious yet mostly in a dumb tabular form. I find myself having intense bursts of concentrated writing and then having to step away from the document  to restore myself both physically and mentally.

I’m currently at the Quality Assurance section of the document, which is in a tabular format to aid both author and reader. However, what I really want to write is this, taken from the Agile Manifesto:

We follow these principles:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity–the art of maximizing the amount
of work not done–is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

I think I will print this on A3 and stick it to our wall.

What attracts me most to Agile methods of software development (I lean towards XP), is the emphasis on human interaction and the focus on values such as trust, respect, simplicity, autonomy and courage. All too often when running a project, the objective of delivering the product dominates and diminishes the creative and social process of producing something that improves our environment.

For me, as Project Manager, the Orbital Project is not only an interesting Research and Development project but also an opportunity to practise a method of human sociability and creativity over a defined period of time. Although I’ve tried to use attributes of Agile methods on projects in the past, this is the first time that I’ve started a project from scratch with this is as the principle method, and a project where I know the Lead Researcher and Lead Developer are likewise keen to work in this way.

This blog is a record of our project over the next 18 months or so. For my part, I’ll be reflecting honestly about the ups and downs of running the project and learning to work closely with people according to the principles quoted above. I’m sure we will fail at times and the process will get lost to the product, but we will learn, even during those times. And gradually, we’ll get better and produce better.

First week meeting notes

Paul and I met earlier to catch up on where we are after the first week of the Orbital project.

We have:

  • Advertised the Lead Developer post. Interviews taking place Wednesday 19th.
  • Completed NOV/JD for Developer post, to be advertised 01/02/12
  • Arranged the Steering Group calendar
  • Met with Jill Stewart, Head of Engineering and most of Project Team for an informal briefing
  • Set up a recurring monthly team meeting
  • Started to put together a reading list
  • Started to arrange for accommodation for the project team
  • Written an article for the Staff Magazine
  • Set up a project website (needs more work)
  • Set up an issue tracker, code repository and project mailing list

We need to:

  • Set up initial meeting with Engineering users
  • Paul to contact Mansur Darlington (ERIM project) and DCC
  • Joss to write Project Plan for 26-10-11
  • Project Website needs work: member bios, press pack, etc. Check given examples.
  • Clarify MRD Programme objectives for everyone at next team meeting,
  • Book conference attendance: USTLG Workshop on research data and Warwick conference. Also, DCC event in Cardiff.
  • Joss to send Simon Hodson, JISC Programme Manager, the project template
  • Lead Developer to join JISCMail when recruited
  • Paul and Joss to book on Programme launch workshop on 1-2/12.

Where appropriate, I’ve added these to our Project tracker.