FAIR Software Checklist

''Does the software have a globally unique and persistent identifier?'' A software object may be assigned with a globally unique identifier such that it can be referenced unambiguously by humans or machines. Globally unique means an identifier should be associated with only one resource at any time. Examples of unique identifiers of data used for software include: Digital Object Identifier (DOI), the Handle System, Uniform Resource Identifier (URI) such as URL and URN, and Software Heritage Identifiers (SWHID). A data repository may assign a globally unique identifier to your data or metadata when you publish and make it available through its curation service. The type of identifier assigned will often depend on the type of repository that the software is deposited in, for example a URL for GitHub, DOI for Zenodo, or SWHID for Software Heritage. Note that URLs are not guaranteed to be persistent and by default GitHub only provides permalinks by request. (checkbox: bind $frsm_01a, "The software has a human and machine-readable unique identifier that resolves to a machine-readable landing page, e.g. a DOI, Handle, SWHID, or URL leading to a landing page") (checkbox: bind $frsm_01b, "The software has a persistant and globally unique identifier in the form of a DOI, SWHID, or ARK") (checkbox: bind $frsm_01c, "The software uses an identifier scheme commonly used in the community the software is used in") (link: "Next metric: FRSM-02 Component Identifiers")[ (if: $frsm_01a)[(set: $f_score to it + 1)] (if: $frsm_01b)[(set: $f_score to it + 0.25)] (if: $frsm_01c)[(set: $r_score to it + 0.25)] (if: $frsm_01a or $frsm_01b or $frsm_01c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-02") ]

''Do the different components of the software have their own identifiers?'' Conceptually, it is useful for identifiers to be assigned at a more granular level than just the software project (often synonymous with the “software concept” or “software project”). For instance a software product may consist of different modules, which in turn may be implemented by different files. This metric tests that these different components are not all assigned the same identifier, and that the relationship between components is embodied in the identifier metadata. The granularity levels for software have been defined by the (link-repeat: "RDA Software Source Code Identifiers WG (p9)")[(open-url: "https://doi.org/10.15497/RDA00053")]. Identifiers for each software component should be globally unique and persistent (as tested by FRSM-01). This metric should not be confused with FRSM-10 and FRSM-12 (related to I2) which checks that other related non-software objects are properly described and FRSM-13 (related to R2) which checks that software dependencies which are not considered a part of the software concept of product are described. This metric is assessed by checking if each software resolves to the appropriate software component, and the identifier metadata is appropriate. (checkbox: 2bind $frsm_02a, "The software has a distinct identifier for each component") (checkbox: 2bind $frsm_02b, "The relationship between component identifiers is embodied in the identifier metadata") (checkbox: 2bind $frsm_02c, "Every component in the software with a granularity level of module or higher has its own identifier") (link: "Next metric: FRSM-03 identifiers for each version")[ (if: $frsm_02a)[(set: $f_score to it + 1)] (if: $frsm_02b)[(set: $f_score to it + 0.25)] (if: $frsm_02c)[(set: $f_score to it + 0.25)] (if: $frsm_02a or $frsm_02b or $frsm_02c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-03") ]

''FAIR Software Assessment'' This website is a prototype of a FAIR self-assessment tool for research software, based on the FAIR-IMPACT metrics for automated FAIR software assessment. The FAIR-IMPACT project has definied 17 metrics that can be used to automate the assessment of research software against the FAIR Principles for Research Software (FAIR4RS), and provides examples of how these might be structured and implemented. The FAIR-IMPACT project is working to implement the metrics as practical tests by extending existing assessment tools such as F-UJI. In the interim, this website tool can be used by research software developers, managers and curators to see how well their software follows the FAIR4RS principles. The information collected by this website is not saved - this tool should only be used as an examplar to help you understand how to use the FAIR4RS principles, assess how well your software currently applies them, and determine ways of improving adoption of the principles. [[Using this tool]] (set: $no_footer to True)

''Using this tool '' This tool was based on the FAIR Principles for Research Software (FAIR4RS) and the FAIR-IMPACT metrics for automated FAIR software assessment. It consists of 17 multiple-choice questions. On answering all the questions, you'll receive a FAIR indicator score. For each question, please check the box for each statement that applies to your software, before moving on to the next metric. A running total of your "score" can be seen at the bottom of the page, Let's get started, by clicking below to go to the question on the first metric. [[FRSM-01 Globally Unique and Persistent Identifiers|FRSM-01]] (set: $no_footer to False)

(if:$no_footer) [  ] (else:) [ (set: $fair_score to $fair_tally / 17 * 100) ---  (meter: bind $fair_score, 100, "X", "FAIRness: (round:$fair_score)", red) (meter: bind $f_score, 11.5, "X", "Findability", green) (meter: bind $a_score, 2.5, "X", "Accessibility", green) (meter: bind $i_score, 4, "X", "Interoperability", green) (meter: bind $r_score, 8, "X", "Reusability", green) ]

(set: $f_score to 0) (set: $a_score to 0) (set: $i_score to 0) (set: $r_score to 0) (set: $fair_score to 0) (set: $fair_tally to 0)

''Does each version of the software have a unique identifier?'' To make different versions of the same software (or component) findable, each version needs to be assigned a different identifier. The relationship between versions is embodied in the associated metadata. What is considered a “version” is defined by the owner of the software: in many cases this will be something that the owner wants to specifically identify and use and/or “release” or “publish” so that others can use and reference/cite. This is something for which there may be disciplinary norms, which may be documented in domain-specific software guidelines. This metric is assessed by checking if each software identifier resolves to a different version, and the identifier metadata is appropriate. (checkbox: 2bind $frsm_03a, "The software has a distinct identifier for each version") (checkbox: 2bind $frsm_03b, "The relationship between identifiers for different versions is embodied in the identifier metadata") (checkbox: 2bind $frsm_03c, "The version number is included in the identifier metadata") (link: "Next metric: FRSM-04 metadata defines purpose")[ (if: $frsm_03a)[(set: $f_score to it + 1)] (if: $frsm_03b)[(set: $f_score to it + 0.25)] (if: $frsm_03c)[(set: $f_score to it + 0.25)] (if: $frsm_03a or $frsm_03b or $frsm_03c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-04") ]

''Does the software include descriptive metadata which helps define its purpose? '' Software requires descriptive metadata to support indexing, search and discoverability. There are several common places for descriptive metadata to be found, including intrinsic metadata that is part of the software source code such as README files, requirements files that describe dependencies, Project Object Model (POM), CodeMeta or Citation File Format (CFF) files, or in the extrinsic metadata available through resolving the software identifier. It may also be directly embedded in software source code files. The implementation of this metric will depend on the coding standards for the programming language as well as community norms for which descriptive metadata is used. This metric can be assessed by checking if the software has machine-readable descriptive metadata that describes its purpose. (checkbox: 2bind $frsm_04a, "The software includes a README or other file which includes the software title and description.") (checkbox: 2bind $frsm_04b, "The software includes other descriptive metadata such as domain, funder, programming language, date created, and keywords.") (checkbox: 2bind $frsm_04c, "The metadata is contained in a format such as CodeMeta or ProjectObjectModel that enables full machine-actionability.") (link: "Next metric: FRSM-05 metadata defines development status")[ (if: $frsm_04a)[(set: $f_score to it + 1)] (if: $frsm_04b)[(set: $f_score to it + 0.25)(set: $r_score to it + 0.25)] (if: $frsm_04c)[(set: $f_score to it + 0.25)] (if: $frsm_04a or $frsm_04b or $frsm_04c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-05") ]

''Does the software include development metadata which helps define its status? '' Software requires descriptive metadata to support indexing, search and discoverability. This includes information that helps identify the current development status of the software. There are many forms of guidance and community standards for structuring development metadata, such as (link-repeat: "RepoStatus")[(open-url: "https://www.repostatus.org/ ")], (link-repeat: "Software Release Practice HOWTO")[(open-url: "https://tldp.org/HOWTO/Software-Release-Practice-HOWTO/index.html")], (link-repeat: "Make a README")[(open-url: "https://www.makeareadme.com/")], and (link-repeat: "AboutCode")[(open-url: "https://www.aboutcode.org/")]. This metric can be assessed by checking if the software has machine-readable descriptive metadata that describes its development and status. (checkbox: 2bind $frsm_05a, "The software includes metadata for contact or support in the README or other intrinsic metadata file according to community standards") (checkbox: 2bind $frsm_05b, "The software includes metadata for development status and links to documentation") (checkbox: 2bind $frsm_05c, "The metadata is contained in a format such as CodeMeta or ProjectObjectModel that enables full machine-actionability.") (link: "Next metric: FRSM-06 metadata defines contributors")[ (if: $frsm_05a)[(set: $f_score to it + 1)] (if: $frsm_05b)[(set: $f_score to it + 0.25)(set: $r_score to it + 0.25)] (if: $frsm_05c)[(set: $f_score to it + 0.25)] (if: $frsm_05a or $frsm_05b or $frsm_05c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-06") ]

''Does the software include metadata about the contributors and their roles? '' Software should make it easy to recognise and credit all contributors. There are several common places for contributor metadata to be found, including README files, CodeMeta or Citation File Format (CFF) files, in the code repository metadata, or in the software identifier metadata. It may also be directly embedded in software source code files. Criteria for which roles are included is normally defined by the community. This metric can be assessed by checking if the software and/or software identifier has machine readable descriptive metadata associated with it that include contributors and roles. (checkbox: 2bind $frsm_06a, "The software includes metadata about the contributors") (checkbox: 2bind $frsm_06b, "The software includes citation metadata that includes all contributors and their roles. This includes ORCIDs when contributors have them.") (checkbox: 2bind $frsm_06c, "The citation metadata includes the proportional credit attributed to each contributor") (link: "Next metric: FRSM-07 software identifier citable")[ (if: $frsm_06a)[(set: $f_score to it + 1)] (if: $frsm_06b)[(set: $f_score to it + 0.25)] (if: $frsm_06c)[(set: $f_score to it + 0.25)(set: $r_score to it + 0.25)] (if: $frsm_06a or $frsm_06b or $frsm_06c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-07") ]

''Does the software metadata include the identifier for the software?'' Software should include its identifier to make it easier to be cited and indexed. There are several common places for identifier metadata to be found, including README files, CodeMeta or CFF files. The choice of location may depend on community standards. This metric can be assessed by checking if the software includes its own software identifier, and that the identifier resolves to that software. (checkbox: 2bind $frsm_07a, "Does the software include an identifier in the README or citation file?") (checkbox: 2bind $frsm_07b, "Does the identifier resolve to the same instance of the software?") (link: "Next metric: FRSM-08 open and persistent metadata record")[ (if: $frsm_07a)[(set: $f_score to it + 1)] (if: $frsm_07b)[(set: $f_score to it + 0.25)] (if: $frsm_07a or $frsm_07b)[(set: $fair_tally to it + 1)] (go-to: "FRSM-08") ]

''Does the software have a publicly available, openly accessible and persistent metadata record?'' Even if the software itself is no longer usable or accessible, its metadata should still be available and accessible. Potential locations for persistent metadata records include scholarly repositories (e.g. Zenodo, HAL, OSF), registries or catalogues (e.g. ASCL, bio.tools, swMath), open scholarly infrastructure (e.g. Wikidata, DataCite, IPOL, eLife). The choice of location is dependent on community standards. This metric can be assessed by checking if the software identifier includes a reference to a persistent landing page or other metadata record, and if that metadata is still accessible. (checkbox: 2bind $frsm_08a, "A metadata record for the software is present on an infrastructure that guarantees persistence.") (checkbox: 2bind $frsm_08b, "The persistent metadata record is available through public search engines. The metadata has a globally unique and persistent identifier.") (checkbox: 2bind $frsm_08c, "The persistent metadata record is available through multiple, cross-referenced infrastructures.") (link: "Next metric: FRSM-09 repository uses standard protocolos")[ (if: $frsm_08a)[(set: $f_score to it + 1)(set: $a_score to it + 1)] (if: $frsm_08b)[(set: $f_score to it + 0.25)(set: $a_score to it + 0.25)] (if: $frsm_08c)[(set: $f_score to it + 0.25)] (if: $frsm_08a or $frsm_08b or $frsm_08c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-09") ]

''Is the software developed in a code repository / forge that uses standard communications protocols? '' Software source code repositories / forges (a.k.a. version control platforms) should use standard communications protocols (such as https / sftp) to enable the widest possible set of contributors. In general, most widely used code repositories / forges use common standardised communications protocols such as https or sftp. In normal use, this test will be implemented by checking that the repository / forge can be accessed using one of these protocols. Using a software forge that is properly indexed by search engines will help with other aspects of findability. This metric can be assessed by checking if the identifier for the code repository / forge can be accessed using standardised communications protocols such as https or sftp. (checkbox: 2bind $frsm_09a, "The code repository / forge can be accessed using the identifier via a standardised protocol.") (checkbox: 2bind $frsm_09b, "If authentication or authorisation are required, these are supported by the communication protocols and the repository / forge.") (link: "Next metric: FRSM-10 open data formats")[ (if: $frsm_09a)[(set: $a_score to it + 1)] (if: $frsm_09b)[(set: $a_score to it + 0.25)] (if: $frsm_09a or $frsm_09b)[(set: $fair_tally to it + 1)] (go-to: "FRSM-10") ]

''Are the formats used by the data consumed or produced by the software open and a reference provided to the format?'' The use of open file formats for data improves the reusability and understandability of the software. This metric is inherently difficult to implement in an automated assessment as at present there is no standardised or common method for describing the data / file formats used by a piece of software in a machine-readable way. Community standards commonly define the data formats in use in a discipline, and resources such as FAIRsharing.org provide a curated catalogues of standards. This metric can be assessed by checking if the software source code and documentation for references to the data formats used. (checkbox: 2bind $frsm_10a, "The documentation describes the data formats used ") (checkbox: 2bind $frsm_10b, "The data formats used are open. ") (checkbox: 2bind $frsm_10c, "A reference to the schema for the data formats is provided.") (link: "Next metric: FRSM-11 use of open, machine-readable APIs")[ (if: $frsm_10a)[(set: $i_score to it + 1)] (if: $frsm_10b)[(set: $i_score to it + 0.25)] (if: $frsm_10c)[(set: $i_score to it + 0.25)] (if: $frsm_10a or $frsm_10b or $frsm_10c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-11") ]

''Does the software use open APIs that support machine-readable interface definition?'' An open Application Programming Interface can be freely accessed by other software or developers, which makes it easier to integrate software and encourages modularity and reuse. The OpenAPI specification is a machine-readable interface definition language for describing, producing, consuming and visualising web services. Additionally, the SmartAPI project has developed a openAPI-based specification for defining the key API metadata elements and value sets, to maximise the FAIRness of web-based APIs. This could be extended to test that the API is callable and does not return an error code. (checkbox: 2bind $frsm_11a, "The software provides documented APIs") (checkbox: 2bind $frsm_11b, "The APIs are open (freely accessible)") (checkbox: 2bind $frsm_11c, "The APIs include a machine-readable interface definition") (checkbox: 2bind $frsm_11d, "No APIs implemented in the software") (link: "Next metric: FRSM-12 References to other objects")[ (if: $frsm_11a)[(set: $i_score to it + 1)] (if: $frsm_11b)[(set: $i_score to it + 0.25)] (if: $frsm_11c)[(set: $i_score to it + 0.25)] ( (if: $frsm_11a or $frsm_11b or $frsm_11c or $frsm_11d)[(set: $fair_tally to it + 1)] (go-to: "FRSM-12") ]

''Does the software provide references to other objects that support its use?'' Determining the usefulness of a piece of software is often aided by understanding what it is used with. This metric is currently difficult to implement as there is no standard machine-readable way to define the relationships at a level of detail that provides suitable meaning, although CodeMeta defines some of these relationships (e.g. supportingData, referencePublication). (checkbox: 2bind $frsm_12a, "The software metadata includes machine-readable references to articles describing the software, articles demonstrating use of the software, or to the data it uses.") (link: "Next metric: FRSM-13 Software requirements")[ (if: $frsm_12a)[(set: $i_score to it + 1)] (if: $frsm_12a)[(set: $fair_tally to it + 1)] (go-to: "FRSM-13") ]

''Does the software describe what is required to use it?'' Software is made more reusable by providing suitable machine-actionable information on dependencies, build and configuration. Most programming languages provide standardised ways of providing dependency information in a machine-actionable format. Build and package management systems can be used to automate the installation process. It is hard to check the relevance / correctness of this information, but it is possible to automatically check for existence and error-free build. Detailed documentation also aids the reusability of software but it is difficult to automatically test for documentation coverage. This metric can be assessed by checking for machine-readable information that helps support the understanding of how it is used. (checkbox: 2bind $frsm_13a, "The software has build, installation and/or execution instructions") (checkbox: 2bind $frsm_13b, "Dependencies are provided in a machine-readable format and the building and installation of the software is automated.") (link: "Next metric: FRSM-14 Test cases")[ (if: $frsm_13a)[(set: $r_score to it + 1)] (if: $frsm_13b)[(set: $r_score to it + 0.25)] (if: $frsm_13a or $frsm_13b)[(set: $fair_tally to it + 1)] (go-to: "FRSM-14") ]

''Does the software come with test cases to demonstrate it is working?'' The provision of test cases improves confidence in the software. Most programming languages have commonly associated test frameworks. The specific definition of what constitutes adequate testing is often defined by community norms. It is hard to check the relevance / correctness of this information, but it is possible to automatically check for existence. This metric can be assessed by checking for the presence of automated tests. (checkbox: 2bind $frsm_14a, "Tests and data are provided to check that the software is operating as expected") (checkbox: 2bind $frsm_14b, "Automated unit and system tests are provided") (checkbox: 2bind $frsm_14c, "Code coverage / test coverage is reported") (link: "Next metric: FRSM-15 Licensing information")[ (if: $frsm_14a)[(set: $r_score to it + 1)] (if: $frsm_14b)[(set: $r_score to it + 0.25)] (if: $frsm_14c)[(set: $r_score to it + 0.25)] (if: $frsm_14a or $frsm_14b or $frsm_14c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-15") ]

''Does the software source code include licensing information for the software and any bundled external software?'' Clear software licensing enables reuse. Each community may have different licences that are popular. It is important that software licences are included with the source code as many tools and processes look for licensing information there to determine licence compatibility. The SPDX License List is a widely used part of the Software Project Data eXchange (SPDX) open standard. Information about the licence for a piece of software can be provided either as a file in the source code repository, or as a short identifier embedded in the source code files. This metric can be assessed by checking the software and its documentation for the presence of a licence. (checkbox: 2bind $frsm_15a, "The software includes its LICENSE file ") (checkbox: 2bind $frsm_15b, "The source code includes licensing information for all components bundled with that software") (checkbox: 2bind $frsm_15c, "The software licensing information is in SPDX format") (link: "Next metric: FRSM-16 Metadata includes licensing information")[ (if: $frsm_15a)[(set: $r_score to it + 1)] (if: $frsm_15b)[(set: $r_score to it + 0.25)] (if: $frsm_15c)[(set: $r_score to it + 0.25)] (if: $frsm_15a or $frsm_15b or $frsm_15c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-16") ]

''Does the software metadata record include licensing information?'' It is important for licensing information to be on the publicly searchable and accessible metadata record. This can be defined in different ways, e.g. the “Rights” field in the DOI metadata. This metric can be assessed by checking if the software identifier or the metadata record referenced by it includes licensing information. (checkbox: 2bind $frsm_16a, "The identifier or metadata record includes licensing and copyright information") (checkbox: 2bind $frsm_16c, "The software licensing information is in SPDX format, or other machine-readable form.") (link: "Next metric: FRSM-17 Provenance information")[ (if: $frsm_16a)[(set: $r_score to it + 1)] (if: $frsm_16c)[(set: $r_score to it + 0.25)] (if: $frsm_16a or $frsm_16c)[(set: $fair_tally to it + 1)] (go-to: "FRSM-17") ]

''Does the software include provenance information that describe the development of the software?'' Good provenance metadata clarifies the origins and intent behind the development of the software, and establishes authenticity and trust. As a type of metadata this overlaps with the metadata called for in guiding principles F2 and F4. It may also be necessary to record information about the way that the software has been developed, such as the development environment used. The methodology for building the software is tested in FRSM-13. It is hard to check the relevance / correctness of this information, but it is possible to automatically check for existence. This metric can be assessed by checking the development metadata available from the code repository / forge for the software. (checkbox: 2bind $frsm_17a, "The software source code repository / forge includes a commit history") (checkbox: 2bind $frsm_17b, "The software source code repository links commits to issues / tickets") (checkbox: 2bind $frsm_17c, "The software project uses other tools to capture detailed machine readable provenance information.") (link: "Finish the assessment")[ (if: $frsm_17a)[(set: $r_score to it + 1)] (if: $frsm_17b)[(set: $r_score to it + 0.25)] (if: $frsm_17c)[(set: $r_score to it + 0.25)] (if: $frsm_17a or $frsm_17b or $frsm_17c)[(set: $fair_tally to it + 1)] (go-to: "Finish the assessment") ]

''Assessment Complete '' You've now considered all the metrics defined by FAIR IMPACT for assessing research software. The overall FAIR score is calculated by identifying what percentage of metrics you ticked at least one checkbox for. These scores are provided only as a way of helping understand adherence to the FAIR4RS principles - they cannot be used for direct comparison between different software. (set: $fair_score to $fair_tally / 17 * 100) (meter: bind $fair_score, 100, "X", "FAIRness: (round: $fair_score)", red) The individual scores for the different categories were: (meter: bind $f_score, 11.5, "X", "Findability: (round: $f_score / 11.5 * 100)%", green) (meter: bind $a_score, 2.5, "X", "Accessibility: (round: $a_score / 2.5 * 100)%", green) (meter: bind $i_score, 4, "X", "Interoperability: (round: $i_score / 4 * 100)%", green) (meter: bind $r_score, 8, "X", "Reusability: (round: $r_score / 8 * 100)%", green) These metrics were developed to be domain-agnostic, and take into account characteristics which are specific to research software such as its executability, its composite nature and its continuous evolution and versioning. Though most of the FAIR4RS Principles can be turned into a measurable metric, some are much harder to quantify, and hence be assessed by any automated tool in the future. In these cases, it may only be possible to test for existence rather than quality or correctness. Others, such as “R3. Software meets domain-relevant community standards” can be seen to apply to many metrics, and the implementation of a metric will reference these community standards. There is an expectation that while the metric and assessment methods will remain the same, the criteria for each compliance level will change as adoption of the FAIR principles increases and infrastructure, tools and guidance improve: what is considered essential should reflect an achievable level of compliance at the current time. [[End]] (set: $no_footer to False)

Thank you for trying out this prototype FAIR Software Assessment tool. (link: "Restart and assess another piece of software")[(restart:)] (link: "Return to the workshop page")[(goto-url: "https://fair-by-design-methodology.github.io/IDCC24workshop/latest/")] (set: $no_footer to False)