| 1 | UCOP Statistics and Analytics - Discussion with Danielle Watters Westbrook Training/webinar on how to get some of the UCOP statistics pulled from Alma | Review and discuss questions related to UCOP Statistics and Analytics with Danielle Update from Working Group re: training/webinar on how to get some of the UCOP statistics pulled from Alma | 50 mins | Danielle / Team | We’ve been working on an Analytics Status Report document that we wanted to send to LG and DOC; originally with an ask to have an exemption or more time on the deadline for gathering statistics and analytics information Danielle is the systemwide library planning analyst at CDL; Danielle or someone else who she works with collect statistics annually for various reporting outputs and obligations Danielle is also the staff analyst for CoUL; in this capacity was previously deployed to help with the SILS project initially doing communications, and then thinking about shared governance Happy to be here to chat both data reporting and the SILS as it relates to analytics
Context about our data obligations, in response to these questions: “Can we secure a recusal from data reporting? If we can’t, what can we do? And what are the consequences of generating imperfect data; data where we know it’s been impacted by issues, x, y, and, z stemming from our recent migration?” UC Libraries have numerous external data reporting responsibilities and obligations; these are predominantly individual campus specific reporting obligations that require some ILS as well as data originating from other local systems such as reference stats, FTE, our spend, digitization, etc. 5 reporting obligations, individual campuses might have some or all of these, and there may be additional reporting obligations that Danielle is not aware of; 5 external obligations: For the statistics collected every Fall, through CDL for the Office of the President (OP), this data is used for a few of these obligations but also overlaps with the other data submissions that follow But specific to what’s shared with CDL in the Fall, Danielle or someone else will process and share components of that data with the UC Liability Office, so it can be shared with our insurer, and importantly this kind of annual reporting activity ensures will all remain in compliance with our insurance policy Other data are summarized at the systemwide level and shared with our Budget Office at their request and their included in the Regents Budget for current operations, which is then published each Fall by the Regents CDL also compiles all of the data and re-shares it back with their campus library stats contacts Campus Libraries also report data to various 3rd parties IPEDS - All campuses report data to the Institute of Education Sciences which is the Statistics and Research arm of the US Department of Education ARL - Most of our campuses are also ARL members (8 of 10) and there are reporting obligations with being an ARL member ACRL - Many if not all also provide data for ACRL Also a lot of overlap between these 3, but particularly between ACRL and IPEDS
It might be clear, but for our insurance policy, this is the most focused data submission that we have; it’s all of our holdings by location These various obligations are managed at the campus level, although CDL does play the facilitation role for submissions to OP For the OP component each year, either Danielle or someone else at CDL does their best to maximize the amount of time that the campuses have to submit data CDL will engage everyone’s campus contacts, which is typically multiple folks at each campus, to understand individual campus timelines, restrictions, and needs
Based on what Danielle has shared with us, is it possible to get an extension or excusal from getting this data? What do we do when we know we’ll have data integrity issues? This is a case by case decision, so it depends; also not for Danielle to dictate, it is up to each campus We have campus experts, assessment librarians, statistics contacts, and together with local leadership that might be engaged, they’ll compile and analyze the data and make a determination on how to best document the issues that impact the data and any steps that might have been taken to address those impacts locally
Speaking generally, data imperfections are incredibly common; our ability to report data is always impacted by the quality of our data and migrations definitely impact that And they also are impacted by the systems we have available to access the data; prior ILS migrations have impacted what and how a campus can pull and report their data Generally, we document issues that are beyond our control In some cases the change in the data is actually an improvement to how we run and analyze the data; so then we document why the change is actually the result of improvements
This year, each campus will take a look at data pulled from the IZ Alma analytics instance, likely compare it to past years, and make a determination on what to submit and how to annotate and document that submission While we definitely have shared data integrity issues this year, Danielle thinks there will likely be varying campus based implications, just how those integrity issues played out in each campus' migration If the IZs are reporting something super wild, the folks managing stats will intervene and make a determination locally on what should be submitted Danielle is happy to be a resource for campuses to share historical data Danielle doesn’t think there has been an instance in a past migration where the issue has been that drastic, but human intervention has definitely occurred when a reporting issue couldn’t be resolved in time for a submission
While these obligations, and given the scale we’re working at, the value in most instances is about the order of magnitude, but it doesn’t always require an exact science
There is a shared desire to move to centralized reporting to save staff time and time across the system, but also to more readily and easily have access to our data Not feasible unfortunately this year within the NZ, but OT and the Ops Center are already working to address this with ExL, and CDL is hiring a systemwide library data focused analyst who will be able to work in collaboration with campus partners and CDL colleagues to provide a more robust systemwide data and reporting service
Similar to what previously-Alma campuses have already done, we just need to do the best we can with the data we have and submit that? Yes, and from an outsider’s perspective, Danielle imagines it would be really valuable to utilize whatever reporting mechanisms or approaches that were used by the campuses that were previously on Alma; acknowledging perhaps imperfections that we have some recently migrated campuses before SILS There have also been time out issues, Danielle has experienced those within the NZ which is super frustrating; some campuses might run into this as well, so it might require further narrowing of scope for a particular query or a conversation if it proves very difficult to get that data out In many ways, it’s a great position to be in, because data is from July 1, 2021 to June 30, 2022, presently we’re not running this data, but if we have those reporting templates, we can begin to play around and we already know what data integrity issues we’re going to come across at a high level We know that format types have been mischaracterized So we can state presently the integrity and computational issues that are impacting how we get the data out or what the data is like when we get it out, and that can inform analyst and assessment folks when they get the data why what they’re receiving looks the way it is
Any additional questions for Danielle? UCOP asks for statistics every year right? And is this what we’re talking about when we we’re asking for an exemption? Yes; the data reported in the Fall to UCOP, in many ways there is overlap with ARL, ACRL, and IPEDS, so in saying we’re not going to do one, this raises the question of whether folks will report any of these Total holdings, reference, ARL includes FTE and things that are not requested by OP; what is not included in later submissions that is included with OP is the request for holdings by location which includes all the sublocations, RLFs, etc.
This makes more sense; since there is so much overlap and there is no getting out of the other reporting things, then what’s the reason to not do the UCOP? And the one remaining one is for the insurance, so we can’t get out of that either This is part of the submission to UCOP, Schedule D Danielle has unsuccessfully lobbied to instead report that detailed data every other year, or can we provide video or some alternate approach so that we don’t have to do that particular data component each year
Schedule D isn’t the challenge; there will be data irregularities based on how the data migrated, but that’s not the schedule that is most concerning Folks in ILL feel incredibly challenged about how to get stats from the AFN; they should open a ticket, but this one feels challenging Some of the schedules don’t probably have an interplay with insurance and don’t have overlap with ARL or IPEDS; could we be successful if we targeted an informational burst that those particular schedules are going to be really out of whack as opposed to other ones that will just be a little bit sketchy? This makes sense, and is good context; Danielle didn’t think holdings by location would be a big pressure point, acknowledging that it will have integrity issues If the in lieu of storage, or shared print, that schedule, ILL might have some overlap
If the time-out issues can’t be resolved with searching systemwide in Alma, does it make sense to think in terms of trying to get a copy of all of our data and put it in another oracle environment that we can search directly against, and possible avoid the time-out issues? Danielle thinks we’ll have to address that issue particularly if the NZ does not prove to be functional At present, Danielle doesn’t know the extent to which it impacts individual IZs; guessing UCLA given its size might have issues with that as well? Yes, Alison Ray pulls data each year around the tier 1 and tier 2 content that CDL helps manage; given the fact that CDL’s migration is behind schedule, Danielle is unsure the origins of that data for pulling it John’s point is really good, there is the near term decision about what we do and if this is a persistent issue, then we need to address it, if not through the vendor then find an alternative means
The equivalent of CDL at the CSUs created several shared analytics reports in the shared zones with its own folder and the report would be like “Schedule A, Question 1,” and the whole report would answer that one question The 23 campuses found this super useful; even though you can’t create a perfect template for every situation, as a starting point it saved a lot of people time Danielle agrees; maybe initially too much time was spent pressing into the NZ to see what was possible to save campuses that time full-fold At present, because the CDL Data Analyst will not be starting until the summer (which was delayed), Danielle agrees this would be a great initial service but it likely wouldn’t be one that CDL can support for this year So it may, in terms of still reducing the workload for many, would be relying on our existing Alma experts, and with many thanks to those 5-campuses that would be willing to share what they can with all of its imperfections and glory
What happens if a campus fails to report to UCOP? This hasn’t happened in the past; we’re one-system and a partnership CDL reaches out in July to start the conversation, to reaffirm that the requirements for reporting haven’t changed in terms of the templates, and ask folks if the timeline will work for them or if there are adjustments CDL can make No clear answer at this point, because this hasn’t been the case previously; guessing it would be a conversation with that campus Not all of that data is required for other reporting, so for the points that are used for ARL and others, could CDL/UCOP get that data if the campus is still reporting to these other groups These other groups though have different due dates; everyone is thinking about UCOP because it comes first and happens in Sept. That’s totally fair; these dates used to be in alignment and they are no longer For the budget office, the earliest deadline is the academic libraries chapter in the regents budget for current operations OP is incredibly proud of the UC Libraries and the SILS and the launch and how we’ve done this all together and that we’re setting ourselves up There was this initial win in migrating together, and the road ahead is then how can we do even more together, provide even better services to our patrons, whether that’s at the campus level or the systemwide level Not submitting data doesn’t emphasize what the SILS has done and what it can do for us
Is there any QA done on the data that is provided in these stats by UCOP? Previously there has been no way for CDL to check source data because ILS’s were entirely at the campus level; but they were processed, reviewed, and followed-up on when certain things seemed off or when schedules didn’t line up CDL shares the data back with all campus contacts to ensure source data is reflected in the submission Nobody is mad if there is a number that is wrong, or seems off
Danielle has an action item to reaffirm the data that is shared with campuses in alignment with the call for stats, that campuses will still receive that CDL component for ARL and ACRL submissions
Thinking more about resource sharing and how that is reported, and considering whether that is potentially a schedule that might be further impacted beyond just data integrity, it seems like it would be good to start that conversation sooner Schedule C that is around deselection due to shared print RLF storage; Danielle can begin asking around to see what the implications are of not collecting that data this year and who might be involved in addressing it or taking ownership For the Resource Sharing part, would be good for Danielle to meet with Caitlin and Alison, since Alison is the Chair of the Resource Sharing group; they could more directly answer Danielle’s questions regarding their needs
The next question for OT - what do we want to submit to LG if anything? Originally our document was asking for excusal or extension; clearly neither of these requests are possible But in our discussion with Danielle yesterday, she recommended that we provide a 3-5 bullet point document that explains what our current path is and the issues that we are seeing, and ultimately explaining that there will be imperfections in our data and probably more so than previous years, so that we set the expectations for those reports What does the rest of the team think about this? Need volunteers to help overhaul the document that we had previously from being an ask to being more of a description that is a bullet point document describing where we’re at and what can be expected of the data and that there will be some imperfections, and send that up to LG
Sarah and Ramon are working on the workshop for the not-already-Alma campuses Update from Sarah: Sarah has folks who are willing to be part of the presentation from UCI, UCR, and UCSC Closing in on June 2nd in the afternoon for the presentation Hoping that if we do this on June 2nd, that will give campuses enough time to implement what we’ll demonstrate by the July 1st date Envisioning going through the UCOP schedule and talking about the Alma based schedules 90-minute session, so there is plenty of time for the presentation and Q&A Will be recorded as well, if this time doesn’t work for folks
Anything that Sarah needs from Danielle for this presentation? Will be nuts and bolts, so the only thing is that if we don’t have to do a certain schedule, then won’t cover it in the workshop Was hard to get folks to be willing to show what they’re doing, because not everyone feels confident in what they’re doing
Danielle thinks there is a broader systemwide conversation about what we collect and to what end and what overlaps with other requirements and what do we actually find super useful for decision making locally and systemwide One thing about the current process with UCOP is that there is an opportunity for each campus to change prior year data to show a difference in the prior year starting value
| | |