Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion db/attachment_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1988,7 +1988,7 @@ func TestWritePathsPreserveUnmigratedAttachmentMetadata(t *testing.T) {
return rev2ID
},
run: func(t *testing.T, ctx context.Context, _ *Database, collection *DatabaseCollectionWithUser, docID, _ string) {
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")
},
Expand Down
33 changes: 29 additions & 4 deletions db/crud.go
Original file line number Diff line number Diff line change
Expand Up @@ -288,8 +288,9 @@ func (c *DatabaseCollection) GetDocChannelHistory(ctx context.Context, docid str
}

// CompactDocChannelHistory removes channel history entries that ended at or before the given sequence number.
// If seq is zero, it removes the history entries of the named channels instead.
// This is used to prune stale channel assignment history to reduce storage overhead.
func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid string, seq uint64) ([]string, error) {
func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid string, seq uint64, channelsToDelete []string) ([]string, error) {
key := realDocID(docid)
if key == "" {
return nil, base.HTTPErrorf(400, "Invalid doc ID")
Expand Down Expand Up @@ -324,25 +325,49 @@ func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid
}

compactedChannels := make(base.Set)
channelsToDeleteMap := make(map[string]struct{})
for _, chanName := range channelsToDelete {
channelsToDeleteMap[chanName] = struct{}{}
}

doc.SyncData.ChannelSetHistory = slices.DeleteFunc(doc.SyncData.ChannelSetHistory, func(channel ChannelSetEntry) bool {
del := channel.End <= seq
var del bool
if seq != 0 {
del = channel.End <= seq
} else {
_, del = channelsToDeleteMap[channel.Name]
}
if del {
compactedChannels.Add(channel.Name)
}
return del
})

doc.SyncData.ChannelSet = slices.DeleteFunc(doc.SyncData.ChannelSet, func(channel ChannelSetEntry) bool {
del := channel.End != 0 && channel.End <= seq
var del bool
if seq != 0 {
del = channel.End != 0 && channel.End <= seq
} else {
_, ok := channelsToDeleteMap[channel.Name]
del = channel.End != 0 && ok
}
if del {
compactedChannels.Add(channel.Name)
}
return del
})

for chanName, chanEntry := range doc.SyncData.Channels {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Avoid compacting Channels entries with Seq==0

SyncData.getCurrentChannels treats both nil removals and non-nil removals with Seq==0 as current channel memberships. CompactDocChannelHistory currently deletes non-nil entries whenever chanEntry.Seq <= seq (seq mode) or when the channel name is listed (channels mode), which can drop a channel that is still considered current and change the effective channel set.

Consider skipping deletions for chanEntry.Seq==0 (consistent with GetDocChannelHistory, which only reports removals where Seq != 0), for example:

Suggested change
for chanName, chanEntry := range doc.SyncData.Channels {
var del bool
if seq != 0 {
del = chanEntry.Seq != 0 && chanEntry.Seq <= seq
} else if len(channels) > 0 {
del = chanEntry.Seq != 0 && slices.Contains(channels, chanName)
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment accurate? That without the check on chanEntry.Seq != 0, this has the potential to remove active channels as well (which we don't want).

Please review and add a test for coverage of this case if we don't already have one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the thing is chanEntry.Seq can never be set to 0? I don't see in any scenario a channel removal will have an entry of 0. Because if a channel is assigned then it will have a non zero value, then on removal I don't see SGW assigning a sequence of 0. The condition that I added in the function that droid pointed out is more of a defensive check.

This will not remove active channels because active channels usually have chanEntry is of type ChannelRemoval, and its nil for active channels.

I modified the test "multiple channels with mixed history" to test this scenario out better. Please let me know if this is okay

if chanEntry != nil && chanEntry.Seq <= seq {
if chanEntry == nil {
continue
}
var del bool
if seq != 0 {
del = chanEntry.Seq <= seq
} else {
_, del = channelsToDeleteMap[chanName]
}
if del {
compactedChannels.Add(chanName)
delete(doc.SyncData.Channels, chanName)
}
Expand Down
2 changes: 1 addition & 1 deletion db/hybrid_logical_vector_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -697,7 +697,7 @@ func TestRestampVersionCASMou(t *testing.T) {
// A second metadata-only write, from a different path - channel history compaction, which unlike
// resync applies to a tombstone as well as a live document. Both previous values have to be
// carried forward, or they stop naming the last write to the body.
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")

Expand Down
2 changes: 1 addition & 1 deletion db/import_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1966,7 +1966,7 @@ func TestMetadataOnlyUpdateWritePaths(t *testing.T) {
return rev2ID
},
run: func(t *testing.T, ctx context.Context, collection *DatabaseCollectionWithUser, docID, _ string) {
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{""})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")
},
Expand Down
39 changes: 30 additions & 9 deletions docs/api/paths/admin/keyspace-_channel_history-compact.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ parameters:
post:
summary: Compact Channel History of Document
description: |-
Compacts channel history for a specified document. Channel history older than the specified sequence will be removed.
Compacts channel history for a specified document. The request must contain either `seq` or `channels`, but not both.

This endpoint removes all channel entries (for sequences before the specified sequence number where the document left the channel),
effectively cleaning up historical channel membership information while preserving active channels and recent changes.
With `seq`, this endpoint removes all channel entries where the document left the channel at or before the specified sequence number.
With `channels`, this endpoint removes the channel history entries for the named channels, whatever sequence the document left them at.

Compaction cleans up historical channel membership information while preserving active channels and recent changes.
This can be useful for reducing metadata size for documents that frequently gain and lose access to channels.

Required Sync Gateway RBAC roles:
Expand All @@ -25,23 +27,42 @@ post:
application/json:
schema:
type: object
required:
- seq
properties:
seq:
description: |-
Channel history having end sequences earlier than this sequence will be removed from the specified document's metadata.
Channel history having end sequences at or before this sequence will be removed from the specified document's metadata.

Cannot be used together with `channels`.
type: integer
format: int64
minimum: 1
example: 12345
channels:
description: |-
Channel history for these channels will be removed from the specified document's metadata.

A non-empty array cannot be used together with `seq`.
type: array
items:
type: string
example:
- channel1
- channel2
oneOf:
- required:
- seq
- required:
- channels
properties:
channels:
minItems: 1
responses:
'200':
description: |-
Successfully compacted channel history from the specified document.
Returns a list of channels that were compacted.

If the response has an empty array, it means either no channels were compacted.
If the response has an empty array, it means no channels were compacted.

content:
application/json:
Expand All @@ -55,13 +76,13 @@ post:
description: |-
Array of channel names that were compacted.
description: |-
A array of all the compacted channels
An array of all the compacted channels
example:
compacted_channels:
- channel1
- channel2
'400':
description: 'Bad request. This could be due to invalid request parameters such as invalid seq value.'
description: 'Bad request. This could be due to invalid request parameters such as an invalid seq value, an empty channels array with no seq, or both seq and a non-empty channels array in the same request.'
content:
application/json:
schema:
Expand Down
Loading
Loading