NIP-44 Geospatial Types #136
Conversation
|
Something @vitorpamplona mentioned on nostr I forgot to consider is that if the content being representing is valid json, then wrapping it in a string adds the overhead of converting it every time. |
|
I am interested in your thoughts about using this approach versus using Geohashing, which seems to be a very simple protocol, geohashes could be just included in any note as a tag and then even indexed automatically by relays (or as @scsibug suggested in the past one could add multiple geohashes with different zoom levels to allow their events to be found by people nearby, if that is an important feature for the given use case). As I see the GeoJSON approach would require custom filtering to be able to fetch things "nearby", but it has also different use cases and benefits in comparison to geohashing (for example, this live location tracking thing that you mention, and separating private events from public events). After some consideration though I feel like we're too far away on client support and network maturity to be able to benefit from these GeoJSON events while geohashing sounds like an easy immediate win (although I wouldn't like to implement it myself either). |
|
NACK
I doubt this and there could be several other ways to misuse it which affects privacy. I don't think we need NIPs to spy on users. Personally, I won't use any client and relay that supports it. |
Hey there @fiatjaf – thank you for the quick response. Geohashing is definitely a simple and efficient solution for points (lat,lon). For developers to be able to support fully featured location-based apps however they’ll need to be able to use the full set of geospatial types (lines, polygons, etc….) I completely agree with you that client and relay support for this is non-existent at this point. I am however working on changing this and have a long list of things that need to be built and tested. The goal is to prove that this works and that people are excited/interested in using it. That’s all on me. The reason for publishing this now was to get things out in the open and raise awareness amongst developers who are skilled and interested in working with this type of data. I did not anticipate it getting approved immediately. In addition to using this as an opportunity to experiment building clients that can interact with event threads that span public/private relays, there’s another really important idea in here. This is the idea of public keys representing physical things. While this has a unique set of data challenges, bringing digital proofs to the physical world would be something else. The use cases around proof of location are plentiful. |
NIPs don't spy on users, companies do. In all seriousness, all this NIP would change about nostr is the idea that if a relay or client gets event with a specific type of 4326 they know what to do with it. Your comment that you wouldn't use a relay or client that supports it is totally acceptable. That's the beauty of all this - freedom of choice, along with transparency and control over your data. I would encourage you to think about this differently though. Imagine a future where you have a nostr client that only connects to relays you own and/or trust. The comments about ad-networks are not throw away lines filled with wishful thinking. I’ve seen it all up-close. With nostr we have a chance to do things differently |
|
Made some updates. I stand behind my opinions, but the point of a protocol is that it does not have an opinion. NIPs should be for the discussion of technical merit. |
|
You clearly have your use case in mind already, but I feel like geo stuff belongs to the tags realm.. these kind 4326 are not searchable or reusable, and only make sense in the context of tagging that event, so why not be a tag directly? that you can attach to any note instead of tagging another event note? |
|
@eskema is making a good point. The entire GeoJSON string can be added as a tag in other events. That is another approach worth considering. I think only specific implementations may give us a good idea of what is the best approach. |
|
Re: searchable content. Geospatial data can be indexed and searched in ways not possible with plain text. You can do things like distance queries, boundary queries, and clustering queries. You would need to store the data in a separate index supported by something like Postgis, but it’s along the same lines if you wanted to build a full text search index in Postgres (or Solr, or Elasticsearch) The thing I feel like I’m not doing a good job explaining is the differences in what you can build when using the tags to store references to locations vs embedding location data in individual notes. This is what jumped out at me when looking at the way that kind 1 notes and their replies work. You are in effect building an emergent, immutable, verifiable, relational database structure. Notes, signatures, and kinds give us the building blocks for the web of structured data where the e and p tags are the new URI. Right now it is messaging between people that's bootstrapping the network, but this will change. @fiatjaf is right though. This is all just theory until someone builds something. I’ll be continuing to work on filling this gap using location as the framework for the model. I appreciate you all taking the time to think about this and providing valuable feedback. |
You have no clue how this NIP alone if implemented could affect nostr. "Use cases" that comprise privacy are normally proposed by governments. I have a few that my government shared while using biometrics. |
|
You are framing this as though it would be required to use, and that it would be public by default. Neither of these are true. I hear where you are coming from, and I think you are looking at it from the wrong angle. If you were to want to share your location with someone privately, how would you do it today? That's where we need to start over, to provide alternatives. If the answer is that you would never share your location with anyone, nothing changes as you would never have to. Reminder, this NIP is about using |
Using nostr DMs or other things |
See, we're on the same team. One day I too would like to share my location with friends and family without involving some random app developer + Apple/Google + carriers. We have to keep this particular thread focused though or people will stumble upon it and wonder how Ross and 1440000bytes got so into the weeds. I hear you though, all feedback is good feedback if we're all focused on making the protocol better. |
There was a problem hiding this comment.
- GeoJSON spec should be linked
- This should be a tag for compatibility with existing kinds
- Privacy implications should be considered
- I think the benefits are kind of BS
- Relay performance: The tag can be extracted and forwarded to an indexer
- "Decentralized data providers" I don't get this.
- "Proof of location" Does it have to be on Nostr? Not everything needs to be on Nostr and you could instead scan a changing QR code at a location which is less vulnerable to spoofing and does not require a new kind
- "Privacy controls": There is cryptographic proof you published a speciifc location and that could have some implications than if there wasn't. Even though this may be routed to private relays this is still an issue
|
Hi @Semisol - thanks for taking on the review and for your feedback. I'll update with the GeoJSON spec. Regarding using a location as a tag vs a new kind. What I've tried to do in the NIP and in this thread is make the case that a physical location is not a tag, it's a valid object type which should be referenced in the same way that a response is to a thread. It's more valuable to the users of the network if data can be linked and not embedded. It makes it consistent and reusable. Location objects would use the same Something which might be helpful is if there was some sort of explanation about what Regular events As for the other objections. My responses are inline below.
Yes, but then relays are responsible for iterating through tags for every event. Top level object identifiers are more efficient. The kind number is there. Why not use it? It seems like a super simple way to describe the content of the note.
Probably a bad term, I'll think on this. Currently authoritative sources of information for things such as business listings are owned by a small number of large companies who are paid to prioritize what's put in front of you. Use pubkeys as location identifiers and smaller communities can build their own directories. Things like user reviews can reference these pubkeys. Anything on the network can be location aware, those locations are referenceable. Don't think about it as a note, think about it as database record. Keys should not be limited to people.
This is a good example. Everything here converges with the combination of the user pubkey, the location pubkey, and whatever the QR code outputs (another pubkey!). You are correct in that not everything needs to live on nostr, but it's nostr that's providing a standard message passing protocol for threading immutable/verifiable data objects in unique and useful ways. In this scenario, all 3 of these things are true in combination. The notes need to travel a path to reach that agreement - it all rides on rails that exist today. Notes are proofs by another name - combined they form more nuanced truths.
Yes. Totally agree. This is true for sensitive location data in the same way that if you say you are going to commit a crime in a |
44.md
Outdated
| @@ -139,6 +139,8 @@ To be clear, this decision is not a technical challenge, but as types in nostr b | |||
|
|
|||
| The most common event on the network is `kind 1` which relays assume will have a string in the `content` field. The `event` object itself is already valid JSON, so allowing for valid JSON to be sent in the `content` would be more efficient for client developers because they would not need to wrap/unwrap and validate JSON content. This could be a complexity trade-off for relays because they would need to support both strings and objects in the content field. Because the `event` must already be valid JSON, and the purpose of the event is that it's entire contents are verifiable by it's signature, it would be reasonable to require relays to support valid JSON in the object field. | |||
|
|
|||
| *update*: Not wrapping the content would break existing code for things such as event signatures and validation. Continuing to treat all `content` as a string and not an object is the only reasonable path forward right now. | |||
There was a problem hiding this comment.
fyi this is unnecessary, there will never be non-string values for content
There was a problem hiding this comment.
Ok not a problem, I've removed it. I originally left it in there as a reference in case someone in the future had a similar question. You are right though in that it's not necessary in the context of this document.
|
Two years ago my company launched a web app that allows people to share their real-time locations in augmented reality and on a 2D map. (The app is yondar.me if you care to see it) Ever since I discovered nostr early last year I've envisioned how Yondar, a centralized location sharing service, could become a decentralized service on an open protocol. I was actually designing my own NIP for geo notes before I ran into this one. But I think I can fold my ideas into what we have going on here. I'll address private location sharing at the end of this post, but I'd like to shift focus to the idea of publishing stuff that you can find on a map in a decentralized way. In my opinion, the purpose of using nostr for geo is to replace Google Maps entirely. Businesses and people should not be slaves to monoliths like Google just to show up on a map and list their store hours. Your nostr map (client) should be your map, and the things that show up on your map are the things you subscribe to, and those things are controlled by the people who own them. Anyone should be able to use this data and display it in a client however they wish: on a 2D map, in augmented reality, in a video game, in a directory, anything! A few considerations and clarifications: Fundamentally, geo information needs to be truthful because it is a claim about the physical world, or the conventional world governed by society, e.g.:
Misleading or inaccurate geo information can be clutter at best and deadly at worst. Therefore, consuming geo information posted to any relay indiscriminately is completely insane. We cannot restrict who publishes geo information, but we can choose who we trust wisely, which is already part & parcel of all responsible nostr activity. I envision that sane people would consume geo notes in 2 ways:
2 allows users to consume large amounts of legitimate geo information from a single source, but it also allows anyone to become a relay operator and make money by helping others get listed. I think of it as breaking the Google Maps:business one:many relationship into a relay:business many:many relationship. Yondar's business model is essentially the paid read-only relay. Businesses pay us to put their places on their own map for their customers to explore in augmented reality. Customers can share their locations while exploring those places and everyone has fun. Why wouldn't people/businesses always just publish their own geo notes for free?
I would like to clarify that geo notes would only seem to be useful for representing geographic data one could not already pull from OpenStreetMaps/Mapbox/Google/etc. Because honestly, nobody needs to worry about adding coordinates for a road or pond into nostr. With few exceptions, all of that geographical detail is covered and catalogued with great accuracy and it would be redundant to replicate it on nostr. Base maps can be pulled into clients easily and cheaply via existing APIs. I know this isn't aligned with chaotic decentralization so if you want to run a road/river/pond relay knock yourself out, but I see no reason to reinvent the wheel on basic, basic maps. That's just the backdrop for what people actually care about, want to find, want others to see. The stuff that really matters to people should be decentralized in nostr and not monopolized in Google's database. Instead of geographic features, I imagine that geo notes would be used mostly to represent things that matter to people, and do it in a decentralized way. Overlay the juicy decentralized stuff on top of the API-served base map in your client. Just in case there are any misconceptions: I do not see geo notes as being used only for businesses. I think they could be used for nearly anything. But businesses work great as a concrete example in this explanation because people want to find them and they are owned literally and cryptographically by an individual. Next, a word about the event definition itself. I agree with @rossbates that the I propose that a geo note would be a replaceable kind (# tbd). Even though I am assuming that geo notes for describing literal geography (ponds, roads) would be mostly useless, literaly geography does change and making it replaceable saves a lot of trouble. The kind of things I assume geo notes would actually be used for, such as food trucks, vendor booths, concerts, parade floats, etc are even more likely to change than physical geography. So replaceability is necessary imho. There was discussion about people sharing locations with each other. I highly recommend that sharing a personal location is a totally different kind:
I think it would be appropriate to have a separate NIP for personal location sharing. It really is a whole different beast from publishing public information about what's out there. Privacy and safety are crucial. Imho there is almost no reason to publish your personal geo coordinates in plaintext for the world to see. Back to geo notes. As far as getting an eventual NIP for geo notes accepted, I am working through the red tape/business side of implementing this NIP into Yondar. Obviously the NIP isn't ready yet, but neither is the red tape. Thanks for considering my thoughts. I am excited to move this forward. |
|
GeoJSON is pretty expressive, but can be too expressive, especially once you get in to polygons with holes and multipolygons. If the primary use case is for points, then the much simpler alternative of a tag like |
I am in favor of simplicity, but the nice thing about GeoJSON is that it can be quite minimal or very expressive, as needed. "Too expressive" reminds me of the ridiculous XML-based geospacial formats. Those are really ugly.
I disagree that the primary use case is for points. I don't think we can predict what the primary use case is for. |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
|
Using a geohash has come up a few times. It would be more efficient for points, but being able to use the entire set of geometry features would dramatically increase the design space for clients/relays. For example, you could use a polygon as a boundary for a building (or town, etc...). Determining if a group of notes are inside/outside of the boundary can be determined by anything, it’s not limited to sources like GPS. Implemented properly, a boundary identified by My preference is for GeoJSON because it’s a web standard supported by many existing libraries. One argument I have against it is actually the flexibility of the properties field. If that field is used to store additional data about the geometry it could lead to a mess. Keeping events as flat as possible encourages re-usability of objects which is optimal for a shared system like this. For example, a business should be a standalone object that references a geometry, it’s not defined by it’s geometry. An alternative solution could be to use something like WKT which wouldn’t allow for anything other than the geometry. This swings towards being very strict. One additional comment on trust and relays. The value of signed events is that you don’t have to trust the relay, only the publisher. I agree that there could be trusted relays in that they are also the source of high-quality data, but optimally the most useful data is spread out across the network and not locked to a relay. Clients can build applications knowing that when they reference an event with a particular identifier they know exactly what’s in it. Object permanence should be considered a strength of this protocol. A pubkey could go rogue tomorrow and start publishing garbage or the relay could disappear, but if all the events published prior were useful, and a client was dependent on that, nothing can undo that. |
|
Thanks to @fiatjaf d82599b now we have a standard list of tags including geohash which will be fantastic for looking up information spacially. I understand your point @rossbates about keeping business information separate from the geometry. Would this then necessitate some new I imagine if a business published a Place note and referenced their brick-and-mortar geo note, and then the business moved, they could replace their Place note with an updated version that didn't reference the old geo note. But then my question is what is the point of having that old geo note at all? It would probably still be crytpographically owned by the business that moved, so it is unlikely that the next business that moves into that building would reference that geo note and risk it being replaced by a new version they don't approve of. I can't think of a reason for bare geo data to simply exist without any metadata/Place information tied to it on nostr. The separation only seems to add useless complexity to how it should all work. I'd like to reiterate that a vast majority of the geometry one might think of — buildings, borders, roads, cities, lakes, etc — doesn not need to be stored in nostr because it exists in public repositories and easy APIs. That is all "base layer" stuff. I think someone should be able to define these things on nostr via geo notes if they want, but I expect that geo notes would more practically be used for stuff people are focused on: marking favorite spots, sharing lists of places, publishing a business listing, or creating geometry for usage in a game. I am working on augmented reality game development and I'm thinking about the fine line between geometry and meshes. I could visibly show geometry from a nostr geo note as a 3D overlay on the real world via your cell phone's camera. More complex meshes should probably be linked as an external asset and loaded rather than storing them in a note. But simpler geo note geometry can also be useful visually in an augmented reality setting. Being able to attach nostr notes to a physical point in space via augmented reality creates some incredible possibilities, especially when you mix in lightning payments. I really want to make this happen. Thanks to all for the helpful discussion so far. So:
|
|
This week Edward Snowden tried nostr and liked the social networking websites/apps although had a few suggestions including privacy: https://iris.to/#/post/858ebc0c5ba79a32914d9760cc46e1c5d07ce6bf4a53caeeba48d44a63e5170b How would this NIP affect such users and nostr in general as protocol if some clients and relays support it because @fiatjaf merged it in this repository ignoring the feedback similar to some other issues and pull requests? I had created an issue related to privacy in which neither @fiatjaf nor @staab was interested to fix it: coracle-social/coracle#3 I had written a blog post about privacy issues in nostr and most of them are still not fixed or even discussed as adding more features seems to be the priority at this moment: https://consentonchain.github.io/blog/posts/nostr-privacy/ I had replied to a tweet about benefits of using encrypted DMs and encrypted channels in different project but suggested to use telegram, Tor etc. although they are useful in many use cases and discussed one recently in electrum wallet repository What are we trying to achieve with this NIP and does anyone involved in nostr protocol development care about privacy? |
I would like to reply to this but I think broader privacy issues shouldn't derail the conversation about this NIP. So I will open a separate issue where I will reply. #190 |
Hello @1440000bytes Your privacy concerns are valid. I think we do need to clarify what this NIP is trying to accomplish. I joined this conversation with my own ideas after @rossbates started it. I figured that discussion here was better than starting a separate geo proposal NIP, so I'm hoping we can all come to some agreement here. Regarding privacy, I'd like to address a few things for you to the best of my ability. TLDR:
Full Explanation: I think it is reasonable to assume that @fiatjaf cares about privacy. Based on that assumption, I presume that the new geohashing tag for events is something that would NOT be utilized unless the user was specifically publishing a note about a location. I don't see any reason why any person would need to publish a note with a geohash of their current location, ever; it's just a bad idea unless you are dying and need help immediately, like a 911-call via note. This is a very absurd use case, so let's assume that practically nobody will ever want to publish their current location. I would also then assume that clients should NEVER publish geohashes of notes unless the geohash was explicitly provided by the user, and it would be wise to suggest that a client should NEVER automatically determine the user's current location geohash. I would infer that the purpose of the geohash tag is so that we can efficiently query relays for notes by physical proximity. The types of notes that would have a geohash tag included would theoretically be notes that represented a public place, like a favorite pub. This Now, users COULD theoretically share their location via nostr, but this location information (in my opinion) MUST be encrypted so that only specific recipients can decrypt it. This means, I think that the geohash tag would NOT be used to share your personal location in this way, and it would be omitted. In my opinion, this NIP is about defining a new Public places visible to everyone have no security concerns, unless people misuse them and publish their home or something like that. But someone could also publish their bank account login via nostr, so you can't prevent people from misusing it. Clients must not abuse geohashing, but there are other things that clients must also not abuse, like receiving your private key! So I don't think there are any significant trust issues in this NIP for defining a standard for creating notes to represent public locations. |
I don't think these assumptions would work as expected. Location of users is being tracked regularly by hackers, 3 letter agencies, Google etc. irrespective of user's permission. Example: https://www.ftc.gov/system/files/documents/public_events/1415032/privacycon2019_serge_egelman.pdf This is the reason some users who are too paranoid about their privacy, remove microphone, gps, camera etc. from the devices they use. Malicious clients/relays could still do this without the NIP and honest client/relays could use location data without NIP as well. Example: This NIP only makes it a standard in the protocol and easier for misuse. I don't think it should exist in a protocol that cares about censorship resistance as good privacy at protocol level makes it difficult for anyone to censor based on different things. |
|
Thanks for the heads up on the When I originally started this NIP it was under the assumption that kinds were for logical object types. Based on this update d82599b it looks like the direction is to embed common objects vs link them. Take the new ‘r’ tag for example. Instead of a kind for URL that enables notes to have a shared identifier, it will require post processing by relays to disambiguate. Everything is a tradeoff, I’m good with it. All that said, someone could also just use the tags to link to other notes using something like nostr:id. This might solve both problems, excited to find out. We just need to all be roughly pointed in the same direction. The one thing the As for the question about GeoJSON vs WKT. I’ll point out that the phrase geohash is being used as the descriptor which is yet a third type of encoding scheme. My personal position now is that As for privacy, there still have been no examples provided of how this would give any organization the ability to extract information from users without their knowledge. Each example use case that requires location data involves parties that are willingly providing it because that’s the whole point of the application. Planes, buses, trains and boats broadcast their location on a constant basis using publicly visible networks. Business put their locations on maps because they want you to know where they are. Concert promoters want you to know an artist is playing at a specific venue. Delivery services need to synchronize location updates between the source, carrier, and destination. Privacy is extremely important, but please stop confusing user privacy with situations where a standardized protocol for broadcasting tamper proof messages would be a perfect fit for objects that are not trying to hide their location, they are actively trying to make their location known. |
So just to be clear, you're saying we could put a geohash, GeoJSON, or WKT inside the g tag rather than the note? And that would be easier to index by relays, so more efficient. Interesting. @1440000bytes if someone wanted to spy on nostr users, they would program their app to get the user's geo coordinates and silently syphon them off to a server somewhere, not even using the nostr network to transmit the data. Any app on your phone can do this to you right now. You are suggesting that because of this NIP, these spies would add a user's location to their own published notes, which would be the most obvious and detectable spying method ever used. People would simply notice this and stop using that client. Any client could theoretically publish more info about you than you want it to. Your client has your private key, it can publish notes whenever it wants. It could fire off a note every 5 seconds with a dump of your clipboard. It could fire off a note every second with your GPS location even without this NIP. People will very quickly stop such a malicious client if it so blatantly and obviously betrays their trust by publishing personal info openly to an open network where everything is visible to everyone, including the victim. But such a malicious client would only have the opportunity to betray a handful of people before nobody ever used it again.
Yes. |
Sure. That's what the other tags being defined in NIPs 12 and 14 are doing. This new approach is treating tags as metadata, the tags don't assume ahead of time that a unique identifier exists for a URL, subject, hashtag, or a geometry. It gains consensus through usage and is only discoverable in aggregate by 3rd party indexes. This is totally fine and within the spirit of nostr. If that's the case, anything representing a geometry can go in the In it's current form the spec is implicitly saying only points can be represented by endorsing a specific encoding scheme (geohash). My take is you can also put polygons in there using GeoJSON or WKT. I mean, why not? There are no restrictions being enforced on Side note: this design does depart from the usage of |
ghost
mentioned this pull request
Awesome. I created a new NIP based on this: #198 |
|
@rossbates This is sounding good. I'd like to dig into the consensus aspect: Anyone can place geometry anywhere. Does consensus in this case mean that when people reference a particular geo note, it gains legitimacy? |
|
@arkin0x That's right, the legitimacy of a location is determined through a combination of the pubkey that created it and how many notes link to it. It's a similar pattern to the rest of nostr, clients can be selective based on the needs and of the application. This lets the definition of "useful" data be defined by those who are using the data, not from a publisher with incentives to distract you. It's an opt-in only system for data sets, with vandalism of existing data prevented by signatures. There's not a minimum threshold for legitimacy either, if I create a location to use in my client then a vote of one is sufficient for my purposes. There's still plenty of room for larger curated data sets from individual publishers, this is what I meant by decentralized data providers. |
|
To make sure we have an understanding: if I am a business and I want to publish my store on nostr because I'm tired of my Google overlords, I can:
And, as a result this NIP isn't strictly necessary anymore. Is this your understanding as well @rossbates? If so, then I think we need a NIP for a |
|
@arkin0x yes I agree there should be a Expanding on the concept. Here's an example of how 5 different nostr types [
|
|
Hi there, I would like to know if it is possible to use this NIP in openstreetmap. Is it possible that this geospecial data will be used here: openstreetmap? |
I've implemented geohashing(g) tags at https://pleb.to/nostrmap One caveat I ran into.. when saving g-tags, you need to consider how clients will search for later. |
I suppose that works and is quick, but also kinda hacky as you're adding redundant tags just to make a 'naive' search work better... more 'proper' would be using a geospatial db (like Tile38 or Redis (Also you probably know this but for context, geohashes share a bit prefix for zoom precision, IIRC it doesn't line up byte-wise so you can't just get the lower resoluyion geohash by recursively cutting off the last char of the higher-res one) |
|
@AngusP you can see it in action https://pleb.to/nostrmap the case where it fails if for searches across low-precision borders (e.g. 9 & c, etc)... but I think its still very usable approach. |
|
@decentropy I've used your geohash technique. It solves a real problem with how relay filters work. If you zoom your map to geohash Applying multiple My implementation is here: https://go.yondar.me Also, I've put my GeoJSON stringified inside the content of a kind 37515 note because the GeoJSON can have additional properties describing the place using conventions from the Google Maps API (like place categorization, etc.) Edit: here is the Yondar repo: https://github.com/innovatario/yondar-mono |
|
@arkin0x BTW, since you mentioned GeoJSON... |
Proposal for a new nostr kind that stores GeoJSON data as the content. This kind, along with the existing p and e tag model, would make notes spatially aware on the network.
https://github.com/rossbates/nips/blob/master/44.md
Looking for comments and feedback. I tried to cover as much as possible, not just how it works but also how it could be used for a wide variety of use cases on both the client and relay side of things.
Something of particular interest in the proposal is the idea of routing notes based on kind to separate public and private data. The protocol is simple enough to handle it, the question is around clients. Would it be easy for users to understand and manage this?