Introduction

Overview

Teaching: 15 min
Exercises: 10 min

Questions

• What does our software need to do to comply with CESSDA’s guidelines?

Objectives

• Explain why a common technical architecture for community development is beneficial.

• Understand the key interoperability characteristics for CESSDA software.

• Awareness of CESSDA’s software development guidelines when developing software.

• Use the CESSDA Software Maturity Levels questionnaire to assess readiness of software for CESSDA.

This episode describes the CESSDA technical architecture, the standards employed by CESSDA, and the maturity levels for software within CESSDA.

Technical Architecture

Having a powerful development infrastructure for community development isn’t enough. It’s also important to ensure that all developers within the community that develop software for the infrastructure are developing it in a consistent and reusable way, and are aware of what is expected of them.

A key reason is to ensure interoperability not only with the technical infrastructure, but also between software components. In the true spirit of community development, a requirement for Service Providers that develop software for CESSDA is that source code remains centrally available to all other Service Providers can access it. This means they can develop successfully against others’ components to ensure this interoperation is successful.

Question: Will it be Reused?

Who has developed software for themselves that later needed to be reused, either by themselves, or by others?

For these reasons, CESSDA has a defined Technical Architecture available that embodies a set of principles, practices and guidelines for developing software for the infrastructure, that covers how to meet CESSDA’s interoperability characteristics and development expectations. Many of these documents are often dry and a hard read, but these architectural guidelines are suitably high-level and a good read for developers, without being particularly restrictive.

The really useful thing that we’ll get to have a hands-on look at later, is their use of an automated build and test system that ensures your code meets common coding standards. Which is great to highlight issues well before deliverable due dates, as well as acting as input to reviews of the software before the code goes into production.

Interoperability Characteristics

The overall software architecture - including software you write that will be deployed on the infrastructure - should meet five common interoperability characteristics. In general, even outside CESSDA, many of these are established best practices you should consider using for any software development.

Loosely coupled but coordinated: enable Service Providers to retain independence, yet fully interact in an integrated service.

Developers just starting out tend towards writing monolithic code that lacks modularisation. But modularisation gives you many advantages to separate your problem into manageable pieces.

Modularisation in Command-line Programs

For a command line program, you can use architectures such as Model View Controller which separate out your program into bits that deal with the core business logic (e.g. simulation code) known as the Model, the bits associated with presenting an interface to users called the Views, and the parts that join these two together, called Controllers.

Essentially, a key modularisation approach in CESSDA is achieved by developing against a microservices architecture exposing a RESTful web service API. Concise but instructive API design guidelines for CESSDA helpfully exists, covering how to structure versioned URL patterns to resources and methods in REST for your service, and how to implement common methods such as querying, filtering, sorting, etc.

With all developers coding to such a defined API style increases consistency between service interfaces and reduces the need for developers to learn a multitude of different APIs for services they wish to use. There are also many resources available online to help you avoid common pitfalls in RESTful API design.

Sustainable: enable medium and long-term investment and business change decisions to be made.

Use of common standards, a common development and test environment, and deployment environment, and a central source code repository will help reduce technical debt (which reflects the increasing costs of additional rework required caused by choosing an easy now solution of a better long-term approach).

Extensible: enable additional services to be built on or around it, including adapting to changing functional requirements over time.

The integration point is the software’s API. New or existing services can be combined as required via their APIs to meet changing functional requirements. APIs should be versioned, and two versions supported simultaneously to allow services to evolve and transition between them.

Maintainable: enable components to be updated when IT specifications change.

Since the integration point is the API, service implementations can evolve as required underneath, such as taking advantage of software or hardware advances.

Standards based: enable the coordinated and planned change to all the coupled, but coordinated, services.

Common architectural standards, particularly a consistent and versioned API, will reduce the burden of change.

Software Development Guidelines

The CESSDA Research Infrastructure provides a ready-made development environment that can be used for free, and will increase the likelihood of software meeting the acceptance criteria. In the grand scheme of getting your software accepted, this means from the outset you can address the acceptance criteria early on in development, which is much easier than trying to address it later when you have a fully formed product.

• Code structure: solid, commented, minimum complexity. Use DRY coding principles to avoid repetition. Adopt suitable coding conventions, use localisation for strings. Configuration should be available thrugh environment variables.
• Environment-specific information: get new developments deployed quickly, keep tools as similar as possible throughout development, testing and deployment environments. Use containers - e.g. Docker.
• Compliance with earlier design decisions: use the Technical Architecture document as the primary guide for informing software development and delivery decisions.
• Documentation throughout development lifecycle: make operational, development, and end-user documentation available in the docs directory of your software. Maintain and update throughout development.
• Source code management: the Bitbucket source control management (SCM) system is mandated.Each development has its own set of repositories within the CESSDA Research Infrastructure project. Each is linked to the CESSDA continuous integration environment, to automate software testing throughout development - we’ll cover these aspects in the next two lessons.
• Security and privacy: a privacy impact assessment is expected against a set of guidelines, particularly to protect personally identifiable information (PII).

Adhering to the above will greatly increase your chances for formal acceptance of your software by CESSDA.

The Golden Rule of Software Development: Write for Someone Else

Of course, there are many ‘Golden’ rules when it comes to software development. But this one is a simple and powerful philosophy that you can apply when developing any code. Whether it be an entire set of sophisticated software tools, or just a simple but useful script.

Essentially, always assume you are writing code for someone else. Assume these others will a) need to read and understand your code, and b) will use that code themselves. For good measure, you could also assume also that they will c) want to modify and extend it. With this mindset, consider how your software should be structured, commented, and documented to achieve these goals. Crucially, that ‘someone’ may be a future version of yourself, perhaps six months from now, who is looking to reuse or extend that code, and needs to understand it again.

Software Maturity Levels

CESSDA has defined a set of Software Maturity Levels, an approach for assessing maturity of software components. This is so CESSDA can mandate minimum levels that SPs and others have to meet - a set of prerequisites for supplying software artefacts to CESSDA:

1. Initial usability; software use is not recommended
2. Use is feasible; the software can be used by skilled personnel but with considerable effort, cost and risk
3. Use is possible by most users; with some effort, cost, and risk. A risk assessment should be made before use
4. Software is usable; with little effort, cost, and risk
5. Demonstrable usability; there is clear evidence that the software is widely used by many users

There’s an online form for assessing 11 criteria:

• Documentation, Intellectual property issues
• Extensibility, Modularity, Packaging
• Portability, Standards compliance, Support
• Verification and testing, Security
• Internationalisation and Localization

Exercise: Evaluate Your (Potential) CESSDA Software!

To familiarise yourselves with the CESSDA maturity levels for software, use the CESSDA Software Maturity Levels questionnaire, assess either software you are in the process of developing for CESSDA, or a piece of software with which you are generally familiar.

The really helpful thing about defining these for contributions is that is gets you thinking early about quality of your software in concrete terms, along a number of dimensions, and how that should apply to your project.

Key Points

• Always develop software according to CESSDA’s Technical Architecture.

• Adhering to the Technical Architecture recommendations will greatly increase the chances that your software is accepted by CESSDA.

• Use the CESSDA Software Maturity Levels questionnaire to assess the readiness of your software for CESSDA.