How successful is my software?

Slides:
https://softwaresaved.github.io/softwaresuccess-rse2017

Notes:
http://bit.ly/softwaresuccess-rse2017

Who I Am

Neil Chue Hong, Software Sustainability Institute

Get these slides

https://github.com/softwaresaved/softwaresuccess-rse2017/

Measuring software is hard

What are we measuring?

quality

engagement

impact

Why are we measuring it?

improvement

comparison

understand role in ecosystem

Introductions

Introductions

Your name

Where you're from

One thing you want to learn from this workshop

... add it to the shared notes ...
http://bit.ly/softwaresuccess-rse2017

Identifying Successful Software

Pair Activity

1. Break into pairs
2. One person in the pair:
• Chooses a piece of research software they think is successful
• Describe to your partner what measures make you think it is successful
• Your partner writes these on a Post-It note as you talk
3. Swap and do the same for the other person in the pair
4. Discuss whether you had the same ways of identifying successful software, and where they differed.

How Do People Identify Success?

Group Discussion

What was important?

What was unexpected?

Does the opposite hold (absence == failure)?

What's easier to measure?

What's more accurate?

... be ready to share ...

Existing approaches

Software Usage

Downloads

Popularity (Forks/Stars)

Citations

Downstream Use

... is someone else using it ...

Software Usability

Functionality

User-friendliness

Performance

Documentation Quality

Multi-platform

... often subjective ...

Project Growth and Maturity

Number of Contributors

Community Activity

Contribution Acceptance

Response Rate

Release Velocity

... visible activity ...

Project Diversity

Contribution diversity

Contributor diversity

Contributor breadth

Decision distribution

Language bias

... hidden indicators ...

Code Quality

Test coverage

Maintainability Indices

Cyclomatic complexity

... can mask other failures ...

Project Risk

Bus factor

Known Vulnerabilities

Bug Age

Downstream Use

... reliability leads to use ...

But ultimately...

High search ranking in Google

Measuring Software

Group Exercise

1. Choose three success criteria that are important to your group
2. Create a small set of metrics that would allow you to rate a piece of software against that measure.
• Apply your metrics to two pieces of software your group is familiar with and compare, OR
• Use these metrics to rank the following set of software:
  • scipy: https://github.com/scipy/scipy
  • Spyder: https://github.com/spyder-ide/spyder
  • pycse: https://github.com/jkitchin/pycse

What are existing models of success?

Many other schemes

See our reference list: https://github.com/softwaresaved/softwaresuccess-rse2017/blob/master/references.md

Contribute via pull requests

Depsy

Depsy: scipy

Depsy: Spyder

Depsy: pycse

Libraries.io and SourceRank

Libraries.io: scipy

SourceRank: scipy

Libraries.io: Spyder

Libraries.io: pycse

CHAOSS Framework

CHAOSS: trends and activity

CHAOSS: dependencies

CHAOSS: comparison

CHAOSS/GrimoireLab: dashboard

What we've learned

What we've learned

Many different ways to measure software

Different people will place different importance
on different measures

Some things are easy to measure,
some things are useful to measure

There are already examples out there -
some may fit your requirements

Contribute your own ideas!