huh.. so not even trying to be compatible with existing hashes?

True. Though if the idea turns out to be better.. then community will adopt it.

if you look at the subject for that twt you will see that it uses the extended hash format to include a URL address.

you rehash the same data too much :P

@bender So turns out something is setting my HashingURI to the value {{ .Profile.URI }} and that is making my hashes wrong so it cannot delete or edit twts.

@movq i'm sorry if I sound too contrarian. I'm not a fan of using an obscure hash as well. The problem is that of future and backward compatibility. If we change to sha256 or another we don't just need to support sha256. But need to now support both sha256 AND blake2b. Or we devide the community. Users of some clients will still use the old algorithm and get left behind.

Really we should all think hard about how changes will break things and if those breakages are acceptable.

I share I did write up an algorithm for it at some point I think it is lost in a git comment someplace. I'll put together a pseudo/go code this week.

Super simple:

Making a reply: 0. If yarn has one use that. (Maybe do collision check?)

1. Make hash of twt raw no truncation.
2. Check local cache for shortest without collision
   • in SQL: select len(subject) where head_full_hash like subject || '%'

Threading:

1. Get full hash of head twt
2. Search for twts
   • in SQL: head_full_hash like subject || '%' and created_on > head_timestamp

The assumption being replies will be for the most recent head. If replying to an older one it will use a longer hash.

These collisions aren't important unless someone tries to fork. So.. for the vast majority its not a big deal. Using the grow hash algorithm could inform the client to add another char when they fork.

@falsifian I believe the preserve means to include the original subject hash in the start of the twt such as (#somehash)

the stem matching is the same as how GIT does its branch hashes. i think you can stem it down to 2 or 3 sha bytes.

if a client sees someone in a yarn using a byte longer hash it can lengthen to match since it can assume that maybe the other client has a collision that it doesnt know about.

@prologic the basic idea was to stem the hash.. so you have a hash abcdef0123456789... any sub string of that hash after the first 6 will match. so abcdef, abcdef012, abcdef0123456 all match the same. on the case of a collision i think we decided on matching the newest since we archive off older threads anyway. the third rule was about growing the minimum hash size after some threshold of collisions were detected.

There is nothing wrong with how we currently run a diff to see what has been removed. if i build a merkle tree off all the twt hashes in a feed i can use that to verify a twt should be in a feed or not. and gossip that to my peers.

So.. basically a rehash of the email "unsend" requests? What if i was to make a (delete: 5vbi2ea) .. would it delete someone elses twt?

isn't the benefit of blake2b that it is a more efficient algo than sha1 and has the same or similar entropy to sha3? i thought we had partially solved this with some type of expanding hash size? additionally we could increase bit density by using base36 or base64/url-safe...

@sorenpeter There was a client that would generate a unique hash for each twt. It didn't get wide adoption.

So this is a great thread. I have been thinking about this too.. and what if we are coming at it from the wrong direction? Identity being tied to a given URL has always been a pain point. If i get a new URL its almost as if i have a new identity because not only am I serving at a new location but all my previous communications are broken because the hashes are all wrong.

What if instead we used this idea of signatures to thread the URLs together into one identity? We keep the URL to Hash in place. Changing that now is basically a no go. But we can create a signature chain that can link identities together. So if i move to a new URL i update the chain hosted by my primary identity to include the new URL. If i have an archived feed that the old URL is now dead, we can point to where it is now hosted and use the current convention of hashing based on the first url:

The signature chain can also be used to rotate to new keys over time. Just sign in a new key or revoke an old one. The prior signatures remain valid within the scope of time the signatures were made and the keys were active.

The signature file can be hosted anywhere as long as it can be fetched by a reasonable protocol. So say we could use a webfinger that directs to the signature file? you have an identity like frank@beans.co that will discover a feed at some URL and a signature chain at another URL. Maybe even include the most recent signing key?

From there the client can auto discover old feeds to link them together into one complete timeline. And the signatures can validate that its all correct.

I like the idea of maybe putting the chain in the feed preamble and keeping the single self contained file.. but wonder if that would cause lots of clutter? The signature chain would be something like a log with what is changing (new key, revoke, add url) and a signature of the change + the previous signature.

# chain: ADDKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w 
# sig: BEGIN SALTPACK SIGNED MESSAGE. ... 
# chain: ADDURL https://txt.sour.is/user/xuu
# sig: BEGIN SALTPACK SIGNED MESSAGE. ...
# chain: REVKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w
# sig: ...

password is generated using caddy hash-password

yarnd does not do auto discovery via webfinger though.. i cant put @username and have it fetch the feed url from webfinger. to fully make feeds portable. would also need to be able to use that for hashing.

@sorenpeter this makes sense as a quote twt that references a direct URL. If we go back to how it developed on twitter originally it was RT @nick: original text because it contained the original text the twitter algorithm would boost that text into trending.

i like the format (<a href="?search=hash" class="tag">#hash</a>) <a href="/timeline/profile?url=url">@nick</a><a href="url" class="webmention"></a> > "Quoted text"\nThen a comment as it preserves the human read able. and has the hash for linking to the yarn. The comment part could be optional for just boosting the twt.

The only issue i think i would have would be that that yarn could then become a mess of repeated quotes. Unless the client knows to interpret them as multiple users have reposted/boosted the thread.

The format is also how iphone does reactions to SMS messages with +number liked: original SMS

@prologic was this in reply to a different thread? Or maybe a hash collision?

I'm not super a fan of using json. I feel we could still use text as the medium. Maybe a modified version to fix any weakness.

What if instead of signing each twt individually we generated a merkle tree using the twt hashes? Then a signature of the root hash. This would ensure the full stream of twts are intact with a minimal overhead. With the added bonus of helping clients identify missing twts when syncing/gossiping.

Have two endpoints. One as the webfinger to link profile details and avatar like you posted. And the signature for the merkleroot twt. And the other a pageable stream of twts. Or individual twts/merkle branch to incrementally access twt feeds.

I have submitted this to be used as the hash tooling for Yarn. See it as a good example on using this in a production environment!

https://git.mills.io/yarnsocial/yarn/pulls/1095

Circling back to the IsPreferred method. A hasher can define its own IsPreferred method that will be called to check if the current hash meets the complexity requirements. This is good for updating the password hashes to be more secure over time.

func (p *Passwd) IsPreferred(hash string) bool {

Hold up now, that example hash doesn't have a $ prefix!

Well for this there is the option for a hash type to set itself as a fall through if a matching hash doesn't exist. This is good for legacy password types that don't follow the convention.

func (p *plainPasswd) ApplyPasswd(passwd *passwd.Passwd) {

@movq You can always use a 5GB video file if the UI hashes it with SHA512 before posting to the server.

@prologic @anth Sounds like a good idea. The hash to conv/search url should stay local to a pod.

@prologic i think i finally suss'd out my hash issue.. now to figure out why im losing avatars on restart.

@prologic yeah it reads a seed file. I'm using mine. it scans for any mention links and then scans them recursively. it reads from http/s or gopher. i don't have much of a db yet.. it just writes to disk the feed and checks modified dates.. but I will add a db that has hashs/mentions/subjects and such.

@prologic I see them is why I ask. like here #cwqmygq they use both hashtag and bangtag?

@prologic Web Key Directory: a way to self host your public key. instead of using a central system like pgp.mit.net or OpenPGP.org you have your key on a server you own.

it takes an email@address.com hashes the part before the @ and turns it into [openpgpkey.]address.com/.well-known/openpgpkey[/address.com]/<hash>

@prologic huh.. true.. the email is md5/sha256 before storing.. if twtxt acted as provider you would store that hash and point the SRV record to the pod. .. to act as a client it would need to store the hash and the server that hosts the image.