Preface
Update February 2022: I have rewritten the code and organized it into a library PersistentHistoryTrackingKit for everyone’s convenience.
I’ve known about the persistent history tracking feature for some time, having browsed the documentation briefly before without taking it too seriously. On one hand, there is not much information about it, making it difficult to learn; on the other hand, I didn’t have a particular impetus to use it.
What is Persistent History Tracking?
Persistent History Tracking is used to determine what changes have been made to a store since that feature was enabled. — Apple Official Documentation
In CoreData, if your data storage format is Sqlite (which most developers use) and you’ve enabled persistent history tracking, any changes in the database (deletions, additions, modifications, etc.), will trigger a system alert notifying apps that have registered for this notification of the change in the database.
Why Use It?
Persistent history tracking currently has the following application scenarios:
-
In the App, merge the data changes generated by the App’s batch operations (BatchInsert, BatchUpdate, BatchDelete) into the current view context (ViewContext).
Batch operations are operated directly through the coordinator (PersistentStoreCoordinator). Since these operations do not go through the context (ManagedObejctContext), if no special treatment is applied, the App will not reflect the data changes caused by batch processing in the current view context in time. Before Persistent History Tracking, we had to merge changes into the context after each batch operation, for example using
mergeChanges. After using Persistent History Tracking, we can consolidate all batch changes into one code segment for merging. -
In an App Group, when an App and an App Extension share a database file, the modifications made by one member in the database are promptly reflected in the view context of another member.
Imagine a scenario where you have an App for aggregating webpage Clips, and you provide a Safari Extension for saving appropriate snippets while browsing web pages. After the Safari Extension saves a Clip to the database, if you bring your App (which has been launched and switched to the background when Safari saves data) to the foreground, the latest Clip (added by the Safari Extension) will not appear in the list. Once Persistent History Tracking is enabled, your App will be notified of the database changes in a timely manner and respond accordingly, allowing the user to see the newly added Clip in the list at the first opportunity.
-
When synchronizing your CoreData database with CloudKit using PersistentCloudKitContainer.
Persistent History Tracking is crucial for synchronizing CoreData with CloudKit data. It is enabled by default when you use PersistentCloudKitContainer as the container, without the need for developers to set it up. However, unless you explicitly enable persistent history tracking in your own code, all data changes synchronized over the network will not notify your code, and CoreData will handle everything quietly in the background.
-
When using NSCoreDataCoreSpotlightDelegate.
At this year’s WWDC 2021, Apple introduced NSCoreDataCoreSpotlightDelegate, which allows CoreData data to be easily integrated with Spotlight. To use this feature, you must enable Persistent History Tracking for your database.
How Persistent History Tracking Works
After enabling Persistent History Tracking for persistent storage, your application will start creating transaction records for any changes that occur in CoreData’s persistent storage. These transactions are meticulously recorded regardless of how they are generated (whether through context or not), or by which App or Extension.
All changes are saved in your Sqlite database file. Apple has created several tables in Sqlite to record information corresponding to the transactions.
Apple has not made the specific structure of these tables public, but we can use the API provided by Persistent History Tracking to query, clear, and perform other operations on the data within.
If you are interested, you can also take a look at the contents of these tables, which Apple has organized very compactly.
ATRANSACTIONcontains the transactions that have not yet been cleared,ATRANSACTIONSTRINGcontains the string identifiers for authors and contextName, andACHANGEis the changed data. This data is ultimately converted into corresponding ManagedObjectIDs.
Transactions are recorded automatically in the order they are generated. We can retrieve all the changes that have occurred after a specific time. You can determine this point in time in several ways:
- Based on a token (Token)
- Based on a timestamp (Timestamp)
- Based on the transaction itself (Transaction)
A basic Persistent History Tracking process is as follows:
- Respond to NSPersistentStoreRemoteChange notifications generated by Persistent History Tracking
- Check if there are still transactions to be processed since the last timestamp
- Merge the transactions that need to be processed into the current view context
- Record the timestamp of the last processed transaction
- Opportunistically delete the merged transactions
App Groups
Before discussing Persistent History Tracking further, let’s first introduce App Groups.
Due to Apple’s strict sandboxing of Apps, each App and Extension has its own storage space and can only read the contents of its sandbox file space. If we want to share data between different Apps, or between an App and an Extension, before App Groups we could only exchange simple data through some third-party libraries.
To solve this problem, Apple introduced its own solution, App Groups. App Groups allow different Apps or an App and an App Extension to share data in two ways (must be under the same developer account):
- UserDefaults
- Group URL (a storage space accessible to every member of the Group)
Most Persistent History Tracking situations occur when App Groups are enabled. Therefore, it is very important to understand how to create App Groups, how to access shared UserDefaults in a Group, and how to read files in the Group URL.
Adding an App to App Groups
In the project navigator, select the Target that needs to join the Group, go to Signing&Capabilities, click +, and add the App Group capability.
Select or create a group in App Groups
The Group can only be added correctly when the Team is set.
The App Group Container ID must start with group. and then usually use the reverse domain name method.
If you have a developer account, you can add App Groups under App ID
Other Apps or App Extensions also specify the same way, pointing to the same App Group.
Creating UserDefaults Shareable within a Group
public extension UserDefaults {
/// UserDefaults for app groups, the content set here can be used by members of the app group
static let appGroup = UserDefaults(suiteName: "group.com.fatbobman.healthnote")!
}suitName is the App Group Container ID you created earlier.
In the code of the App in the Group, the UserDefaults data created using the following code will be shared by all members of the Group, and each member can read and write to it:
let userDefaults = UserDefaults.appGroup
userDefaults.set("hello world", forKey: "shareString")Obtaining the Group Container URL
let groupURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.com.fatbobman.healthnote")!Operating on this URL is exactly the same as operating on the URLs within the App’s own sandbox. All members of the Group can read and write to files in this folder.
The following code assumes that the App is in an App Group and shares data through UserDefaults and Container URL.
Enabling Persistent History Tracking
Enabling the Persistent History Tracking feature is quite straightforward. We simply need to configure the NSPersistentStoreDescription.
Here is an example of enabling it within the CoreData template Persistence.swift generated by Xcode:
init(inMemory: Bool = false) {
container = NSPersistentContainer(name: "PersistentTrackBlog")
if inMemory {
container.persistentStoreDescriptions.first!.url = URL(fileURLWithPath: "/dev/null")
}
// Add the following code:
let desc = container.persistentStoreDescriptions.first!
// If desc.url is not specified, the default URL is the Application Support directory of the current app
// FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
// Enable Persistent History Tracking on this Description
desc.setOption(true as NSNumber,
forKey: NSPersistentHistoryTrackingKey)
// Enable remote change notifications
desc.setOption(true as NSNumber,
forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)
// The configuration for the description must be completed before loading, otherwise it will not take effect
container.loadPersistentStores(completionHandler: { (storeDescription, error) in
if let error = error as NSError? {
fatalError("Unresolved error \(error), \(error.userInfo)")
}
})
}If creating your own Description, similar code would be as follows:
let defaultDesc: NSPersistentStoreDescription
let groupURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "group.com.fatbobman.healthnote")!
// The database is saved in the App Group Container, which can also be accessed by other Apps or App Extensions
defaultDesc.url = groupURL
defaultDesc.configuration = "Local"
defaultDesc.setOption(true as NSNumber, forKey: NSPersistentHistoryTrackingKey)
defaultDesc.setOption(true as NSNumber,
forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)
container.persistentStoreDescriptions = [defaultDesc]
container.loadPersistentStores(completionHandler: { _, error in
if let error = error as NSError? {
}
})The Persistent History Tracking feature is set on the description. Therefore, if your CoreData uses multiple Configurations, you can enable this feature only for the configurations that need it.
Responding to Persistent History Tracking Remote Notifications
final class PersistentHistoryTrackingManager {
init(container: NSPersistentContainer, currentActor: AppActor) {
self.container = container
self.currentActor = currentActor
// Register to respond to StoreRemoteChange
NotificationCenter.default.publisher(
for: .NSPersistentStoreRemoteChange,
object: container.persistentStoreCoordinator
)
.subscribe(on: queue, options: nil)
.sink { _ in
// The content of the notification is meaningless, it simply serves as a signal that something needs to be processed
self.processor()
}
.store(in: &cancellables)
}
var container: NSPersistentContainer
var currentActor: AppActor
let userDefaults = UserDefaults.appGroup
lazy var backgroundContext = { container.newBackgroundContext() }()
private var cancellables: Set<AnyCancellable> = []
private lazy var queue = {
DispatchQueue(label: "com.fatbobman.\(self.currentActor.rawValue).processPersistentHistory")
}()
/// Process persistent history
private func processor() {
// Operate in the correct context to avoid affecting the main thread
backgroundContext.performAndWait {
// fetcher is used to obtain transactions that need processing
guard let transactions = try? fetcher() else { return }
// merger merges the transactions into the current view context
merger(transaction: transactions)
}
}
}Let me briefly explain the code above.
We register the processor to respond to NSNotification.Name.NSPersistentStoreRemoteChange.
Whenever there is a data change in your database for an Entity with Persistent History Tracking enabled, the processor will be called. In the code above, we completely ignore the notification because its content is meaningless; it merely tells us that the database has changed, and the processor needs to handle it. What exactly changed, whether it needs to be processed, etc., must be determined through our own code.
All operations related to Persistent History Tracking data are performed within backgroundContext to avoid affecting the main thread.
PersistentHistoryTrackingManager is the core of our handling Persistent History Tracking. In the CoreDataStack (such as the above-mentioned persistent.swift), you can handle Persistent History Tracking events by adding the following code in init:
let persistentHistoryTrackingManager : PersistentHistoryTrackingManager
init(inMemory: Bool = false) {
....
// Mark the author name of the current context
container.viewContext.transactionAuthor = AppActor.mainApp.rawValue
persistentHistoryTrackingManager = PersistentHistoryTrackingManager(
container: container,
currentActor: AppActor.mainApp // the current member
)
}Since members of the App Group can read and write to our database, we need to create an enumeration type to tag each member in order to better distinguish which member generated the Transaction in subsequent processing.
enum AppActor: String, CaseIterable {
case mainApp // iOS App
case safariExtension // Safari Extension
}Create tags for the members as per your requirements.
Fetching Transactions that Need Processing
After receiving the NSPersistentStoreRemoteChange message, we should first extract the Transactions that need processing. As mentioned earlier in the working principle, the API provides us with 3 different methods:
open class func fetchHistory(after date: Date) -> Self
open class func fetchHistory(after token: NSPersistentHistoryToken?) -> Self
open class func fetchHistory(after transaction: NSPersistentHistoryTransaction?) -> SelfFetch Transactions after a certain point in time that meet the conditions
Here I recommend using Timestamp, that is, Date, for processing for two main reasons:
- When we use UserDefaults to save the last record,
Dateis a structure directly supported by UserDefaults, requiring no conversion. Timestampis already recorded in the Transaction (tableATRANSACTION), which can be directly found without the need for conversion, whereas Token requires additional calculation.
By using the code below, we can obtain all the Transaction information in the current sqlite database:
NSPersistentHistoryChangeRequest.fetchHistory(after: .distantPast)This includes Transactions from any source, regardless of whether they are necessary for the current App or whether they have already been processed by the current App.
In the processing flow mentioned above, we have introduced the need to filter unnecessary information through timestamps and save the timestamp of the last processed Transaction. We save this information in UserDefaults for easy handling by members of the App Group.
extension UserDefaults {
/// Get the latest timestamp from all app actors' last timestamps
/// Only delete transactions before the latest timestamp to ensure that other appActors
/// can normally obtain unprocessed transactions
/// Set a 7-day limit. Even if some appActors do not use (no userdefaults created),
/// it will retain transactions for no more than 7 days
/// - Parameter appActors: app roles, such as healthnote, widget
/// - Returns: Date (timestamp), nil return value will process all unprocessed transactions
func lastCommonTransactionTimestamp(in appActors: [AppActor]) -> Date? {
// Seven days ago
let sevenDaysAgo = Date().addingTimeInterval(-604800)
let lasttimestamps = appActors
.compactMap {
lastHistoryTransactionTimestamp(for: $0)
}
// All actors have no set value
guard !lasttimestamps.isEmpty else {return nil}
let minTimestamp = lasttimestamps.min()!
// Check if all actors have set values
guard lasttimestamps.count != appActors.count else {
// Return the latest timestamp
return minTimestamp
}
// If it has been more than 7 days without receiving values from all actors, return seven days to prevent some actors from never being set
if minTimestamp < sevenDaysAgo {
return sevenDaysAgo
}
else {
return nil
}
}
/// Get the timestamp of the last processed transaction for the specified appActor
/// - Parameter appActor: app role, such as healthnote, widget
/// - Returns: Date (timestamp), nil return value will process all unprocessed transactions
func lastHistoryTransactionTimestamp(for appActor: AppActor) -> Date? {
let key = "PersistentHistoryTracker.lastToken.\(appActor.rawValue)"
return object(forKey: key) as? Date
}
/// Set the latest transaction timestamp for the specified appActor
/// - Parameters:
/// - appActor: app role, such as healthnote, widget
/// - newDate: Date (timestamp)
func updateLastHistoryTransactionTimestamp(for appActor: AppActor, to newDate: Date?) {
let key = "PersistentHistoryTracker.lastToken.\(appActor.rawValue)"
set(newDate, forKey: key)
}
}Since every member of the App Group saves their own lastHistoryTransactionTimestamp, to ensure that the Transaction can be correctly merged by all members before being cleared, lastCommonTransactionTimestamp will return the latest timestamp of all members. lastCommonTransactionTimestamp will be used when clearing merged Transactions.
With these foundations, the above code can be modified to:
let fromDate = userDefaults.lastHistoryTransactionTimestamp(for: currentActor) ?? Date.distantPast
NSPersistentHistoryChangeRequest.fetchHistory(after: fromDate)With the timestamp, we have already filtered a large number of Transactions that we do not need to worry about, but are all the remaining Transactions what we need? The answer is no. At least there are two types of Transactions we do not need to concern ourselves with:
-
Transactions generated by the current App’s own context
Typically, the App will immediately respond to changes in data generated through the context. If the change is already reflected in the view context (main thread ManagedObjectContext), then we can ignore these Transactions. However, if the data is completed through batch operations or operations in
backgroudContextand has not been merged into the view context, we still need to process these Transactions. -
Transactions generated by the system
For example, when you use PersistentCloudKitContainer, all data synchronization over the network will generate
Transactions, which will be handled by CoreData, and we do not need to concern ourselves with them.
Based on the above two points, we can further narrow down the range of Transactions that need processing. The final code for fetcher is as follows:
extension PersistentHistoryTrackerManager {
enum Error: String, Swift.Error {
case historyTransactionConvertionFailed
}
// Get filtered Transactions
func fetcher() throws -> [NSPersistentHistoryTransaction] {
let fromDate = userDefaults.lastHistoryTransactionTimestamp(for: currentActor) ?? Date.distantPast
NSPersistentHistoryChangeRequest.fetchHistory(after: fromDate)
let historyFetchRequest = NSPersistentHistoryChangeRequest.fetchHistory(after: fromDate)
if let fetchRequest = NSPersistentHistoryTransaction.fetchRequest {
var predicates: [NSPredicate] = []
AppActor.allCases.forEach { appActor in
if appActor == currentActor {
// This code assumes that in the App, even operations performed through backgroud have been immediately merged into ViewContext
// Therefore, for the current appActor, only process transactions generated by the batchContext context
let predicate = NSPredicate(format: "%K = %@ AND %K = %@",
#keyPath(NSPersistentHistoryTransaction.author),
appActor.rawValue,
#keyPath(NSPersistentHistoryTransaction.contextName),
"batchContext")
predicates.append(predicate)
} else {
// For transactions generated by other appActors, all need to be processed
let predicate = NSPredicate(format: "%K = %@",
#keyPath(NSPersistentHistoryTransaction.author),
appActor.rawValue)
predicates.append(predicate)
}
}
let compoundPredicate = NSCompoundPredicate(type: .or, subpredicates: predicates)
fetchRequest.predicate = compoundPredicate
historyFetchRequest.fetchRequest = fetchRequest
}
guard let historyResult = try backgroundContext.execute(historyFetchRequest) as? NSPersistentHistoryResult,
let history = historyResult.result as? [NSPersistentHistoryTransaction]
else {
throw Error.historyTransactionConvertionFailed
}
return history
}
}If your App is relatively simple (for instance, not using PersistentCloudKitContainer), you may not need the more refined
predicateprocessing procedure described above. In general, even if the obtainedTransactionsexceed the required scope, the pressure on the system caused by CoreData during merging is not significant.
Since fetcher further filters Transactions through NSPersistentHistoryTransaction.author and NSPersistentHistoryTransaction.contextName, please clearly mark the identity in your NSManagedObjectContext code:
// Mark the context's author in the code, for example
viewContext.transactionAuthor = AppActor.mainApp.rawValue
// If used for batch processing, please mark the name, for example
backgroundContext.name = "batchContext"Clearly marking Transaction information is a basic requirement for using Persistent History Tracking.
Merging Transactions into the View Context
After fetching the necessary Transactions with a fetcher, we need to merge these Transactions into the view context.
The merging operation is quite simple; after merging, just save the last timestamp.
extension PersistentHistoryTrackerManager {
func merger(transaction: [NSPersistentHistoryTransaction]) {
let viewContext = container.viewContext
viewContext.perform {
transaction.forEach { transaction in
let userInfo = transaction.objectIDNotification().userInfo ?? [:]
NSManagedObjectContext.mergeChanges(fromRemoteContextSave: userInfo, into: [viewContext])
}
}
// Update the last transaction timestamp
guard let lastTimestamp = transaction.last?.timestamp else { return }
userDefaults.updateLastHistoryTransactionTimestamp(for: currentActor, to: lastTimestamp)
}
}You can choose the merging code according to your preference. The following code is equivalent to the NSManagedObjectContext.mergeChanges shown above:
viewContext.perform {
transaction.forEach { transaction in
viewContext.mergeChanges(fromContextDidSave: transaction.objectIDNotification())
}
}These Transactions that have occurred in the database but have not yet been reflected in the view context will be immediately reflected in your App UI after merging.
Cleaning Up Merged Transactions
All Transactions are stored in the Sqlite file, which not only occupies space but also affects the access speed of Sqlite as the records increase. We need to establish a clear cleaning strategy to delete the processed Transactions.
Similar to the fetcher using open class func fetchHistory(after date: Date) -> Self, Persistent History Tracking also provides us with three methods for cleaning tasks:
open class func deleteHistory(before date: Date) -> Self
open class func deleteHistory(before token: NSPersistentHistoryToken?) -> Self
open class func deleteHistory(before transaction: NSPersistentHistoryTransaction?) -> SelfDelete Transactions that occurred before a specified point in time and meet the conditions.
The cleaning strategy can be rough or fine. For example, Apple’s official documentation adopts a rather rough cleaning strategy:
let sevenDaysAgo = Date(timeIntervalSinceNow: TimeInterval(exactly: -604_800)!)
let purgeHistoryRequest =
NSPersistentHistoryChangeRequest.deleteHistory(
before: sevenDaysAgo)
do {
try persistentContainer.backgroundContext.execute(purgeHistoryRequest)
} catch {
fatalError("Could not purge history: \(error)")
}This deletes all Transactions that occurred 7 days ago, regardless of their author. In practice, this seemingly rough strategy rarely causes any problems.
In this article, we will treat the cleaning strategy more finely, just like we did with the fetcher.
import CoreData
import Foundation
/// Deletes processed transactions
public struct PersistentHistoryCleaner {
/// NSPersistentCloudkitContainer
let container: NSPersistentContainer
/// app group userDefaults
let userDefault = UserDefaults.appGroup
/// all appActors
let appActors = AppActor.allCases
/// Cleans up processed persistent history transactions
public func clean() {
guard let timestamp = userDefault.lastCommonTransactionTimestamp(in: appActors) else {
return
}
// Get the request for deletable transactions
let deleteHistoryRequest = NSPersistentHistoryChangeRequest.deleteHistory(before: timestamp)
// Only delete Transactions generated by members of the App Group
if let fetchRequest = NSPersistentHistoryTransaction.fetchRequest {
var predicates: [NSPredicate] = []
appActors.forEach { appActor in
// Clean up Transactions created by App Group members
let predicate = NSPredicate(format: "%K = %@",
#keyPath(NSPersistentHistoryTransaction.author),
appActor.rawValue)
predicates.append(predicate)
}
let compoundPredicate = NSCompoundPredicate(type: .or, subpredicates: predicates)
fetchRequest.predicate = compoundPredicate
deleteHistoryRequest.fetchRequest = fetchRequest
}
container.performBackgroundTask { context in
do {
try context.execute(deleteHistoryRequest)
// Reset the timestamp for all appActors
appActors.forEach { actor in
userDefault.updateLastHistoryTransactionTimestamp(for: actor, to: nil)
}
} catch {
print(error)
}
}
}
}The reason I set such detailed predicates in both fetcher and cleaner is that I use the Persistent History Tracking feature within PersistentCloudKitContainer. CloudKit synchronization generates a large number of Transactions, thus a more precise filtering of the operation target is necessary.
CoreData will automatically handle and clear Transactions generated by CloudKit synchronization, but if we accidentally delete CloudKit Transactions that have not yet been processed by CoreData, it may lead to database synchronization errors, and CoreData may clear all current data and attempt to reload data from remote.
Therefore, if you are using Persistent History Tracking on PersistentCloudKitContainer, be sure to only clear Transactions generated by members of the App Group.
If you are only using Persistent History Tracking on PersistentContainer, there is no need for such thorough filtering in both fetcher and cleaner.
After creating PersistentHistoryCleaner, we can choose the timing of the call based on our actual situation.
If using PersistentContainer, you can try a more aggressive cleaning strategy. Add the following code in PersistentHistoryTrackingManager:
private func processor() {
backgroundContext.performAndWait {
...
}
let cleaner = PersistentHistoryCleaner(container: container)
cleaner.clean()
}This way, after each response to the NSPersistentStoreRemoteChange notification, an attempt will be made to clear the merged Transactions.
However, I personally recommend a less aggressive cleaning strategy.
@main
struct PersistentTrackBlogApp: App {
let persistenceController = PersistenceController.shared
@Environment(\.scenePhase) var scenePhase
var body: some Scene {
WindowGroup {
ContentView()
.environment(\.managedObjectContext, persistenceController.container.viewContext)
.onChange(of: scenePhase) { scenePhase in
switch scenePhase {
case .active:
break
case .background:
let clean = PersistentHistoryCleaner(container: persistenceController.container)
clean.clean()
case .inactive:
break
@unknown default:
break
}
}
}
}
}For example, perform the cleaning when the app goes to the background.
Summary
The complete code for this article can be downloaded from Github.
The following resources have been crucial for this article:
-
This book by Donny Wals is one of my favorite recent books on CoreData. It has a chapter on Persistent History Tracking. His Blog also frequently features articles about CoreData.
-
Avanderlee’s blog also has a lot of great articles about CoreData. The article Persistent History Tracking in Core Data provides very detailed explanations. The structure of the code in this article is also influenced by it.
Apple built Persistent History Tracking to allow multiple members to share a single database while keeping the UI up to date. Whether you’re building a suite of applications, want to add the right Extension to your App, or just respond to batch operation data in a unified way, Persistent History Tracking can provide you with good assistance.
While Persistent History Tracking may impose a slight system burden, this is negligible compared to the convenience it offers. In practical use, I hardly notice any performance loss due to it.