Scorecard Job Visibility

Hey @bradley , You are right that currently we do not show job status or execution logs for Scorecard.  There are 3 main jobs here - Resolver , Snapshot and Batch job that runs if you make any config changes and chose to run it right after. I will add this to our backlog but having an execution log in place for scorecards would take longer and is not on this FY23 roadmap.

One thing we already planned and are implementing in the coming quarter is that while one of the jobs is in progress , we will float a banner on the admin page about the job run, advising not to make any changes while the job is in progress to scorecard config. 
In addition to that, from your Idea, we will try to capture the failed records , if any, into the Batch completion email to fill the gap.  

That would be super helpful, as the completion emails aren’t reliable either, so any incremental improvement to this would be great. It would also be helpful if support was aware they could find out on their end that the job is completed and report back if it was, when asked.

Doing some scorecard replacement this morning and the lack of completion notifications or status checks really anywhere makes it difficult to confirm whether reassignments have been completed etc.

@ChiragJ @revathimenon 👀?

Checking with the team internally @bradley 

Hi @bradley @TMaier , we implemented this in the product already-  When one of the jobs is in progress , we float a banner on the admin page about the job run, advising not to make any changes while the job is in progress to scorecard config. 
Apart from this, we currently don’t have plans to add job status into the product. 

I appreciate the incremental improvement, but that addresses like 1% of the ask here. From what you’re saying, the only plan is a one time per run banner shows up saying the job started, and that’s it? Does it go away when the job is done? Does it just go away when you get rid of it regardless of how the job is running/ran?

What we’re asking for is:

• Current Job status (init, in progress, aborted, failed, etc)
• General execution history of a given job run (e.g., what did it do and to what records)
• Overall execution history

None of the actual ask will be addressed?

I’d like to see a screenshot of the expected banner as well - I can’t say this has caught my eye since it was added, and I still regularly wonder whether a scheduler batch job is complete or not.

This is definitely not behaving the way I expected it to based on the description above. I’ve just updated a Scorecard (added a Measure) and the only indication that a job is running is a very short-lived pop-up error when I try to save additional changes. See screenshots below…

I am currently waiting on 3 different scheduler jobs after having updated assignment criteria on some scorecards, and I’m not seeing any banner or other indication that the jobs are running.

@TMaier , @bradley  The banner shows upfront when the Resolver job is running and it shows once you are within a scorecard’s configuration screen. Refer article here

For batch jobs triggered upon updating configuration (measure or measure weight changes) of a scorecard, we currently don’t have a banner but we show a toast message when you try to save a configuration as you rightly captured. We can try to get this as a banner similar to resolver but this may not be on our near time roadmap. 

ChiragJ wrote:

@TMaier , @bradley  The banner shows upfront when the Resolver job is running and it shows once you are within a scorecard’s configuration screen. Refer article here

For batch jobs triggered upon updating configuration (measure or measure weight changes) of a scorecard, we currently don’t have a banner but we show a toast message when you try to save a configuration as you rightly captured. We can try to get this as a banner similar to resolver but this may not be on our near time roadmap. 

@ChiragJ I think what we are trying to convey, is that the alleged banner is so narrowly available as to not be useful (if it even appears), and it does not really address the elements we are asking for.

What we’re asking for is:

• Current Job status (init, in progress, aborted, failed, etc)
• General execution history of a given job run (e.g., what did it do and to what records)
• Overall execution history

First time ever seeing this banner, and I just did a bunch of scorecard criteria update work last week.

@bradley , I understand the ask and it will help you get more visibility into the job execution , but can you help me with any other use cases I may be missing. When we did the enhancement to get the banner (part of another feedback), we wanted customers to avoid making changes during the execution runs and not be able to save it. 

@ChiragJ I think it would be interesting to hear what it is you understand from what @TMaier and I have been saying in our comments. Because it very much seems like we’re talking past each other at this point, and my guess is that we’re getting lost in semantics.

I understand your comments about the banner to mean that for the Resolver job specifically, which by default runs at 23:59, this banner will appear. This is the only action that will trigger the banner, and it will not show for any other reason. It is unclear to me if “scorecard scheduler” is the same thing as the resolver job in this context, but I am inferring that they are the same thing.

Speaking for myself, I am looking for execution history, once again, as outlined by these items here:

• Current Job status (init, in progress, aborted, failed, etc)
• General execution history of a given job run (e.g., what did it do and to what records)
• Overall execution history

As an example, have a look at Horizon Rule execution history for a given rule, or even Connector Job execution logs. That is the type of functionality and specificity I am looking for.

In terms of context, or when I would to know when an execution is ongoing, stalled, or completed would not be limited to the Resolver job. It would include the resolver job, but it would be any time a scorecard change is make as is being applied.

For example:

Yesterday, I made some scorecard changes. I did NOT see the alleged banner. I did, however, receive this email after some time after the change:

Again, when I made another change, the banner was not there. Again, I received an email like this sometime later:

I think the banner addresses maybe 10% of the use case, and execution logs would have covered about 99%.

Hopefully this helps clarify some of what I mean. If anything on the use case is not clear, I’m happy to chat about it in a call :)

​@ChiragJ any thoughts here on my last comment?