Additional Metadata such as latest version number, date, and software risk

[slominskir]

It may be useful to add other metadata such as latest version and latest release date.

Other fields such as # of lines of code could help with comparing maintenance burden of software.

[slominskir]

Kyle is suggesting an "operational impact" field. Or perhaps a complexity ranking. Discussed briefly ideally software is weighted somehow - as not all software is equal.

[khesse-757]

Add Risk Assessment Fields

Add metadata fields to assess and prioritize software based on risk. A calculated Risk Score helps identify which software needs the most attention for documentation, cross-training, and ownership planning. Adapted from the Certified software risk assessment before Chris retired.

New Numeric Fields

Field	Description	Values
Size	Scope of the software (lines, components, how big)	1 (small) – 5 (large)
Volatility	How often this software changes	1 (stable) – 5 (frequent changes)
Complexity	Technical complexity and learning curve	1 (simple) – 5 (very complex)
Ops Criticality	Impact on operations if broken	1 (minimal) – 5 (ops stops)
Infrastructure Criticality	How many other systems depend on this	1 (standalone) – 5 (many dependents)
Depth	How many people can maintain this	1 (3+ people) – 5 (1 person or less)

Calculated Field

Field	Description	Calculation
Risk Score	Overall risk	Sum of all fields (6–30)

New Text Fields

Field	Description	Example Values
Documentation Status	Quality of documentation	Comprehensive / Good / Sparse / Minimal / None
Special Access	Users, accounts, or host access required to maintain/deploy	sqam, vxuser, batman, accadm, cedadm, root, database access, specific hosts

UI

• Numeric fields and Risk Score sortable in main table
• Filter by Risk Score range (show me everything 20+)
• Filter by Depth (show me everything 1-deep)
• Filter by Documentation Status (show me Sparse or None)
• Filter by Special Access (show me anything requiring sqam)
• Maybe color coding Risk Score: 6-12 (green), 13-20 (yellow), 21-30 (red)

Use Case

• "Which software should we prioritize for documentation and cross-training?"
• "What's our most fragile software?" (high Depth score)
• "If we cross-train one more person on X, what's the risk reduction?" (Depth goes from 5 ->3)
• Helps ACG answer: "If we have limited time and resource, which software should we prioritize?"

Progress

Things like when cross-training happens the Risk Score drops.

[slominskir]

We could use the weighted categories to inform a software downtime risk matrix predicting the likelihood and impact of software downtime. I'm assuming the biggest risk to ACG is software not working as expected (unavailable), so that would be the risk we are tracking. This is most important during machine run periods, but extends to SAM (non-machine running) periods as well.

Impact Ratings:
Software with a possibility of machine damage or personnel injury would be critical impact (5). Software that completely blocks the program (run period) or completely blocks SAM activities would be high impact. Partial/Degraded program / SAM activities would be medium. Software that creates a nuisance situation but otherwise can be worked around would be marked as low impact. Software that doesn't affect ops perhaps is marginal (1). We could directly assign impact 1-5 to each software record:

• Ops Impact (risk impact)

Probability Rating (derived from weighted categories):
Probability (likelihood) would take into account the weighted categories. I think documentation status would be a numeric weighted category as well. All of the numeric weighted categories can be Enum values (have labels). Documentation status is probably one of the more important weighted categories. Size is really part of complexity so there might be a roll-up situation that feeds into complexity.

Ops Criticality maps directly to impact in the risk matrix. Not just about dependencies (standalone vs many dependents) as alluded, though that can be part of the calculus for impact. It's probably really about program fully blocked or degraded and/or critical if personal injury or machine damage.

Weighted Categories

• Code Complexity (likelihood we can understand it easily!)
• Documentation Quality (likelihood we can find answers quick!)
• Cross Training (likelihood we can find an expert quick!)

Code Complexity - lines of code, programming language (C++ memory management), subject domain (physics!), dependencies (number, complexity), organization/structure/style (spaghetti code!). Objective measures are best. We can measure lines of code, number of unique paths through functions (cyclomatic), and count operators and operands (halstead metrics).

Documentation Quality - we can try to be objective about it by using language specific tool use quality and look for a set of docs including (README, API DOCS, userguide, extra docs: on-call notes or technical notes in wiki or similar format). A 1 is awarded if little to no docs. A 2 if at least one of the doc types is present with decent quality. A 3 if two decent quality documents exist. A 4 if three decent quality docs exist. A 5 if four or more decent quality docs exist.

Cross Training / Depth - Objectively could simply be the number of people whom are experts (can be expected to fix a problem). It's probably more complicated than that though as the target is likely 2 experts and three tangentially familar staff. So this metric may not be linear, a 2 means two experts, but a 3 could mean 2 experts and 1 familiar staff. We don't expect every ACG staff member to be an expert in everything as that would be spread too thin. The balance is at least one cross trained person (in addition to software primary dev) and then others whom are familiar with technologies and subject domain used by software.

I'm not sure we want volatility. I think the implied impact may often be reverse of alluded - things that change frequently may actually be more reliable. In my experience software atrophies over time and when I come back to software I haven't touched recently it's using out-dated language features and libraries and there is a context switch to re-familiarize myself with it and often relearn pecularities. Frequently updated software part of continous integration (always kept in working state), may be less risky than something we haven't touched in ages. However, things that have continued to work without being touched are great too - the problem is someone may upgrade something it depends on and it breaks and we are now flat-footed. Maybe call this metric a wash and just leave it out?

[slominskir]

There is perhaps another weighed category to consider: debuggability: how easy is software to debug. Is there a log file? How easy to observe what software is doing? Is code modular and easy to isolate issues? Are specialized tools required to debug? Does the software require beam studies to debug? Are there unit/integration tests? This metric may inform how long the downtime will be more so than how frequently there will be downtime (though it could be a factor in frequency as well).

Many of the categories actually affect frequency and duration of downtime so I suppose we need to be careful when discussing risk matrix which we mean (or if both consider very carefully implications of downtime duration vs frequency).

[slominskir]

Another category that could roll-up into downtime probability could be empirical/qualitative measure of past performance. How often has the software caused downtime in the past? Could be more valuable than trying to guess how fragile software is based on complexity, documentation, debuggability, and number of experts.

[khesse-757]

Good points. Agree on dropping volatility (cuts both ways), rolling size into complexity, and merging the criticality fields into impact.

What do you think of this:

Fields

Field	Description	Values
Complexity	Technical complexity (includes size, language, dependencies, structure)	1 (simple) – 5 (very complex)
Documentation	Quality and completeness of docs	1 (Comprehensive) – 5 (None)
Depth	Cross-training coverage	1 (2+ experts, others familiar) – 5 (1 person, no backup)
Impact	Consequence if unavailable	1 (nuisance) – 5 (ops stops / safety / can't do work)

Complexity probably needs more discussion. I like the enum idea, it brings clarity to the numbers.

Impact Scale

Value	Label	Description
1	Marginal	Can easily workaround
2	Low	Nuisance, workaround exists
3	Medium	Degraded ops or SAM activities
4	High	Blocks program or SAM activities
5	Critical	Ops stops, machine damage, personnel safety

Debuggability

Interesting. Knowing how easy the software is to diagnose and fix is good. Randy would use MTTF/MTTR kind of metrics for other categories. Could argue it's wrapped into depth and documentation though? Debuggability also varies by who's doing the debugging. For example, a low-level app developer fixing an IOC problem or someone from ACE fixing a certificate issue. Based on who is doing the task will change the recovery time based on experience, documentation, etc.

If I were to put debugging into a table, perhaps something like this:

Value	Label	Description
1	Easy	Logs, tests, modular, standard tools, documented workflow, any user
2	Manageable	Some logging, reasonably isolated, known process
3	Moderate	Limited observability, some specialized knowledge, limited documentation
4	Difficult	Sparse logging, tightly coupled, specialized tools, special permission/user
5	Very Difficult	No logs, requires beam studies, no tests, black box, little/no documentation

Data

It's great if we can pull downtime history from DTM. But it really only gets captured when we're running and skews toward ops criticality. Like operators are not logging a downtime for ATLis being unavailable. Feels like many of our metrics will be hand maintained. Not sure we have the infrastructure in place to automatically calculate impact, weight and downtime probability.

Next

After talking with Scott, we should get a few more people on board. Maybe meet when Theo gets back. Having a clear goal will help focus scope. I see this as a way to track ACG software inventory, do some risk assessment, prioritize documentation/cross-training, and reduce single points of failure. Answer questions like "where are we fragile?" and "where should we put effort?"

[slominskir]

Yeah, I think we're starting to converge. One thing we may consider is flipping metrics around and using qualifiers or antonyms so they all align with 1 = robust/best/few issues, 5 = fragile/worst/many issues. I think this means:
Depth = Esoterism
Documentation = Documentation Gaps
Debuggability = Debug/Testing Difficulty

This way we can average those weighted categories and they align.

I started to write this down in directory user guide to think it through. We can edit as needed:
https://acgdocs.acc.jlab.org/en/software-directory-guidance#software-risk-guidance

As far as past downtime rate, we can just manually provide it for now. I'm thinking none of the weighed categories would be automatically set initially, maybe version 2.0 we can try to query repo for lines of code and query DTM for downtime counts, etc. Perhaps we default all values as 3 (middle ground), and then only if software is extra bad or good in one of the categories do we have to change anything. Pull Request demonstrating rough draft of these metrics incoming.

[slominskir]

The scope of this issue is now limited to risk assessment fields. We can create a new issue if we decide to add version info and release date to SIM. Starting to look like that's less useful as users can just navigate through to upstream repo to get that info.

Staging the Pull Request on acctest for prototype evaluation. We can create additional issues to fix up the prototype.