Tackling the large contact list problem

(this was sent via email but also sent to the github issue as a comment)

Re: [damus-io/damus] Bug: stale feed (Issue #2194)
Link: https://github.com/damus-io/damus/issues/2194

On Sat, May 04, 2024 at 07:09:46PM GMT, alltheseas wrote:
>reduced relays - does not change stale feed
>[..]
>Turned on damus on old phone - same issue present

Your contact list is large and contains lots of hashtags and pubkeys.
This may have something to do with it.

We should use these to build some test cases for this to dig down and
see what is going on.

I have some suspicions:

## Potential problems

- Your contact list is about 150KB, subscribing to that 9x is about 1.4MB
of upload every time your phone reconnects. The sheer weight of this is
likely causing stalling.

- Relays may reject large subscriptions like this, leading to empty feeds.

- There may be some data encoding problem in the list leading to a
   rejected subscription, but I did not see any issue on your contact
   list. I did see some lists like this though.

## Options

- Outbox model to only send subsets of your list to the sets of relays
   that your follow list is on. This is probably the best path without
   writing any new NIPs but may lead to missed messages depending on how
   damus chooses which relay subsets to read from for each pubkey subset.

   If most users have the damus relay on their list we may be back to the
   same problem. Relay reliability is also a concern in this scenario as
   well as all the other outbox model issues in Damus which still needs
   to be resolved by the local relay model[1][2]

- Only choose a subset of the relays to send the timeline subscription
   to. This may result in missed messages.

## Workarounds

Since your contact list is so large, you may need to either:

- Start going through and unfollowing people who are no longer active

or

- Severely reduce your relay list to 1-4 until we have a fix.

This is a huge problem with the protocol at the moment with not many
obvious solutions. I try to keep my contact list under 1000 for this
reason.

We should definitely run some simulations on how much data the outbox
model will save us here, with real world data from real relay and
contact lists such as yours.

## Next steps

- Look into ways for damus to suggest stale accounts to unfollow

- Put together potential outbox model subscription selection algorithms
   and see how much it helps us (may only help with a diverse selection
   of relays on relay lists)

- Use elat's current contact list as a testbed for fixing this issue
   since it is an example testcase that is not working.

- Look into follow caps to reduce the damage of this issue but this may
   be controversial, perhaps we can find a solution without doing this.

## Ideal solution

- Look into a potential new NIP for relays. Relays already have your
   contact list, what if the query was simple as:

   { "authors": ["kind3:<pubkey>"] }

   Which would do the same thing as the 150k query except just tell the
   relay to use the list it has instead of sending it over and over.

   Then we could send a few bytes instead of 150k * number of relays

   I prefer this approach, not sure if anyone has proposed it yet.


I have a feeling a lot more users are going to be running into this
issue so we should starting thinking of solutions soon.

Cheers,

	Will


[1] https://github.com/damus-io/damus/issues/2041
[2] https://github.com/damus-io/notedeck/

Received on Sunday, 5 May 2024 18:42:28 UTC