macadmin.fraserhess.com

Howard Oakley posted this weekend about problems installing XProtect updates when a Content Caching server is available. His post prompted me to dig into a similar situation I had last week: I couldn’t install Command Line Tools for Xcode while my Content Caching service was online.

Fortunately, /var/log/install.log had a lot of verbose logging about the failed installation including the URLs used to fetch packages from the Caching service. There in the log was a smoking gun (emphasis mine):

2023-01-30 15:37:58-07 b21222 softwareupdated[518]: 012-92431: Encountered Invalid Data for CLTools_macOS_SDK.pkg with URL http://192.168.2.47:42578/content/downloads/03/28/012-92431-A_FKICGWU4EK/eflw1v4c64sgmvux4ljc083cfjj663wu9g/CLTools_macOS_SDK.pkg?source=swcdn.apple.com&sourceScheme=https

The full URL made it easy to download the suspect package from the Caching server. The downloaded package had some problems. Its signature could not be verified nor could it be opened/parsed by Suspicious Package. file(1) confirmed it was a xar archive as expected but due to the other evidence, my diagnosis was that the cached package was corrupt. 

I reconstructed the original URL and downloaded the source package to compare. Not only did the two packages have different checksums (openssl md5), the one from Apple opened in Suspicious Package, and its signature could be verified. Diagnosis confirmed.

There’s no good way to purge corrupt assets in Content Caching. It’s something of an interminable problem. Assets are removed in 3 ways:

1. All assets are purged by an administrator
2. An asset becomes the least recently used and is purged when the cache needs to make space for new assets
3. Apple updates the source file with a new modified date. (The Caching service checks for an updated modification date every time an asset is requested.)

Purging all assets is the nuclear option. It’ll remove the corrupt asset, sure, but also every valid asset. The other two aren’t really options admins can execute — they are just how Content Caching internally removes and updates assets. If we want to remove a single asset, we’ll need another way.

An alternative approach is to remove the asset from Content Caching’s SQLite database. The script below takes Content Caching down, edits the database, and brings Content Caching back up. When Caching comes back up, it scans its data directory and purges the now orphaned files from the disk.

The script requires a URI to look up the asset in the database. As the script will take Content Caching down temporarily, be careful not to interrupt connected clients.

#!/bin/bash
# Purge1Asset.sh
#
# Takes a URI and purges it from a local Content Caching server

if [ "$(id -u)" -ne 0 ]; then
    echo "Run script as root/sudo"
    exit 1
fi

if [ -z "$1" ]; then
    echo "Pass a URI "
    echo "$0 "
    exit 1
fi

DATA_PATH=$(defaults read /Library/Preferences/com.apple.AssetCache.plist DataPath)
DATA_PATH="${DATA_PATH:-/Library/Application Support/Apple/AssetCache/Data}"
DB_PATH="$DATA_PATH/AssetInfo.db"

if ! [ -f "$DB_PATH" ]; then
    echo "Caching database not found"
    exit 1
fi

AssetCacheManagerUtil deactivate

sqlite3 "$DB_PATH" "delete from zasset where zuri='$1';"

AssetCacheManagerUtil activate

I ran this with the following URI: /content/downloads/03/28/012-92431-A_FKICGWU4EK/eflw1v4c64sgmvux4ljc083cfjj663wu9g/CLTools_macOS_SDK.pkg

When Content Caching restarts, it logs that it removed the associated files from disk:

Cache manager removed orphaned asset directory 270493E7-E048-40E8-A9E8-C498FD34585E
Cache manager removed 1 orphaned asset directory(ies)

With the corrupt asset purged, I downloaded the asset using the Content Caching URL from above. The package was no longer cached, so Content Caching contacted Apple’s server and downloaded a fresh copy. The refreshed package could be validated and its checksum matched the package downloaded directly from Apple. After this, I was able to successfully install Command Line Tools for Xcode with Content Caching online.