神楽坂しえる @Clworld@md.ggtea.org

(あたまびみょうにいたい)

わわわー

ねこねこ

にゃー

ぐぬぬ https://md.ggtea.org/media/rgT3-J1AHXdUFBGINm4

test3

?????

Most recent statuses on timeline are already arriving out of order even if we are using sequential id for status. we may already lost some statuses when we fetch statuses via REST API...

sequence id is numbered when INSERT SQL executed..
INSERTed row become visible from other transaction when it's COMMITed..
Mastodon does .. 1. create status(INSERT) 2. process tag, emoji, mentions, attachments 3. COMMIT. ... There is long time gap between INSERT and COMMIT.

INSERTする時にidが生成されるのってINSERTを発行した時だろうし、それが公開されるのはCOMMITした時だよねぇ…。多分.. ついでにMastodonはstatusを作った後にタグとか絵文字とか画像添付を見ている訳で1msどころではなくめっちゃ時間ずれてる気がする。

@aschmitz @unarist @Gargron I rethinking that and found that there is time gap between status INSERT and transaction COMMIT. Depending when ids are numbered.., I'm wondering now that there is some lost when fetching statuses via REST API already?

ももももーにん

にゃーん にゃーん にゃーん

@aschmitz @unarist @Gargron Note: We should taking care about API should return statuses as many as DEFAULT_STATUSES_LIMIT when there is more statuses in range when filtering redis result. Returning statuses count < DEFAULT_STATUSES_LIMIT has meaning there is no more statuses in requested range. When removing max_id and since_id, 2 more statues we should get from redis.

@aschmitz @Gargron @unarist
I only try to clarify problem.
Problem 2 is not a problem for current PR's implementation. I think.
'sequence data' part of id have enough randomness and ids with same 53bit prefix does not happen.
Some tries to make 'sequence data' sequencial will affected by problem 2. If same 53bit prefix occurs, some problem occurs I think.

@unarist @aschmitz @Gargron I found some misunderstanding about max_id and since_id, API doesn't return max_id and sice_id itself..
API returns ids smaller than max_id and not return max_id... returning same msec id will confrict with them...

… ？？？

@Gargron @aschmitz @unarist
There are 2 problems.
1. Id non-sequentiality within msec. It causes lack of status when getting new status.
2. Redis key precision problem
Additional filtering for redis result can solve 2, but can't solve 1.
Returning same msec statuses at range border can solve both 1 and 2, but It causes status dup rarely and client should dedup them.
Public page should do some additional filtering for dedup (I think public page does not have 'read new' function).

@unarist @aschmitz @Gargron
Users of REST client(which does not do dedup) may see duplication of status rarely when there is statuses which is created within same msec.
Using sequential id only for redis may works but feel some difficulty and may incompatible with backfilling.

@Gargron @aschmitz @unarist I guess what it means:
Current snowflake id which Mastodon uses does not sequential within msec.
So, Don't return same msec status may cause lack of status within msec when some cases in paging.
Returning same msec status help clients for getting all status, but it causes duplication of status when there is statuses which created in same msec.
So, client should perform dedup and ignore statuses already got when calling REST API.