Updated version

Announce stream improvements

MIRROR.md

@@ -0,0 +1,33 @@
+This repository is a public mirror. All potential future development is happening elsewhere.
+
+I am stepping back from all public-facing interaction with this project. Reticulum has always been primarily my work, and continuing in the current public, internet-facing model is no longer sustainable.
+
+The software remains available for use as-is. Occasional updates may appear at unpredictable intervals, but there will be no support, no responses to issues, no discussions, and no community management in this or any other public venue. If it doesn't work for you, it doesn't work. That is the entire extent of available troubleshooting assistance I can offer you.
+
+If you've followed this project for a while, you already know what this means. You know who designed, wrote and tested this, and you know how many years of my life it took. You'll also know about both my particular challenges and strengths, and how I believe anything worth building needs to be built and  maintained with our own hands.
+
+Seven months ago, I said I needed to step back, that I was exhausted, and that I needed to recover. I believed a public resolve would be enough to effectuate that, but while striving to get just a few more useful features and protocols out, the unproductive requests and demands also ramped up, and I got pulled back into the same patterns and draining interactions that I'd explicitly said I couldn't sustain anymore.
+
+So here's what you might have already guessed: I'm done playing the game by rules I can't win at.
+
+Everything you need is right here, and by any sensible measure, it's done. Anyone who wants to invest the time, skill and persistence can build on it, or completely re-imagine it with different priorities. That was always the point.
+
+The people who actually contributed - you know who you are, and you know I mean it when I say: Thank you. All of you who've used this to build something real - that was the goal, and you did it without needing me to hold your hand.
+
+The rest of you: You have what you need. Use it or don't. I am not going to be the person who explains it to you anymore.
+
+This is not a temporary break. It's not "see you after some rest", but a recognition that the current model is fundamentally incompatible with my life, my health, and my reality.
+
+If you want to support continued work, you can do so at the donation links listed in this repository. But please understand, that this is not purchasing support or guaranteeing updates. It is support for work that happens on my timeline, according to my capacity, which at the moment is not what it was.
+
+If you want Reticulum to continue evolving, you have the power to make that happen. The protocol is public domain. The code is open source. Everything you need is right here. I've provided the tools, but building what comes next is not my responsibility anymore. It's yours.
+
+To the small group of people who has actually been here, and understood what this work was and what it cost - you already know where to find me if it actually matters.
+
+To everyone else: This is where we part ways. No hard feelings. It's just time.
+
+---
+
+असतो मा सद्गमय
+तमसो मा ज्योतिर्गमय
+मृत्योर्मा अमृतं गमय

README.md

@@ -1,5 +1,7 @@
 # Nomad Network - Communicate Freely
 
+*This repository is [a public mirror](./MIRROR.md). All development is happening elsewhere.*
+
 Off-grid, resilient mesh communication with strong encryption, forward secrecy and extreme privacy.
 
 ![Screenshot](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1.png)
@@ -148,14 +150,17 @@ You can help support the continued development of open, free and private communi
   ```
 - Bitcoin
   ```
-  bc1p4a6axuvl7n9hpapfj8sv5reqj8kz6uxa67d5en70vzrttj0fmcusgxsfk5
+  bc1pgqgu8h8xvj4jtafslq396v7ju7hkgymyrzyqft4llfslz5vp99psqfk3a6
   ```
 - Ethereum
   ```
-  0xae89F3B94fC4AD6563F0864a55F9a697a90261ff
+  0x91C421DdfB8a30a49A71d63447ddb54cEBe3465E
   ```
+- Liberapay: https://liberapay.com/Reticulum/
+
 - Ko-Fi: https://ko-fi.com/markqvist
 
+
 ## Development Roadmap
 
 - New major features
@@ -171,8 +176,6 @@ You can help support the continued development of open, free and private communi
     - Better navigation handling when requests fail (also because of closed links)
     - Retry failed messages mechanism
     - Re-arrange buttons to be more consistent
-    - Input field for pages
-    - Post mechanism
     - Term compatibility notice in readme
     - Selected icon in conversation list
     - Possibly a Search Local Nodes function

nomadnet/Directory.py

@@ -7,6 +7,7 @@ import threading
 import RNS.vendor.umsgpack as msgpack
 
 from LXMF import pn_announce_data_is_valid
+from nomadnet.util import strip_modifiers
 
 class PNAnnounceHandler:
     def __init__(self, owner):
@@ -229,7 +230,7 @@ class Directory:
 
     def display_name(self, source_hash):
         if source_hash in self.directory_entries:
-            return self.directory_entries[source_hash].display_name
+            return strip_modifiers(self.directory_entries[source_hash].display_name)
         else:
             return None
 
@@ -241,7 +242,7 @@ class Directory:
                 if dn == None:
                     return RNS.prettyhexrep(source_hash)
                 else:
-                    return dn+" <"+RNS.hexrep(source_hash, delimit=False)+">"
+                    return strip_modifiers(dn)+" <"+RNS.hexrep(source_hash, delimit=False)+">"
             else:
                 return "<"+RNS.hexrep(source_hash, delimit=False)+">"
         else:
@@ -250,13 +251,13 @@ class Directory:
                 if dn == None:
                     return RNS.prettyhexrep(source_hash)
                 else:
-                    return dn
+                    return strip_modifiers(dn)
             else:
                 return "<"+RNS.hexrep(source_hash, delimit=False)+">"
 
     def alleged_display_str(self, source_hash):
         if source_hash in self.directory_entries:
-            return self.directory_entries[source_hash].display_name
+            return strip_modifiers(self.directory_entries[source_hash].display_name)
         else:
             return None
 

nomadnet/_version.py

@@ -1 +1 @@
-__version__ = "0.9.2"
+__version__ = "0.9.5"

nomadnet/ui/textui/Browser.py

@@ -8,9 +8,11 @@ import shutil
 import nomadnet
 import subprocess
 import threading
+from threading import Lock
 from .MicronParser import markup_to_attrmaps, make_style, default_state
 from nomadnet.Directory import DirectoryEntry
 from nomadnet.vendor.Scrollable import *
+from nomadnet.util import strip_modifiers
 
 class BrowserFrame(urwid.Frame):
     def keypress(self, size, key):
@@ -36,19 +38,20 @@ class BrowserFrame(urwid.Frame):
                     if hasattr(self.delegate, "page_pile") and self.delegate.page_pile:
                         def df(loop, user_data):
                             st = None
-                            nf = self.delegate.page_pile.focus
-                            if hasattr(nf, "key_timeout"):
-                                st = nf
-                            elif hasattr(nf, "original_widget"):
-                                no = nf.original_widget
-                                if hasattr(no, "original_widget"):
-                                    st = no.original_widget
-                                else:
-                                    if hasattr(no, "key_timeout"):
-                                        st = no
-                        
-                            if st and hasattr(st, "key_timeout") and hasattr(st, "keypress") and callable(st.keypress):
-                                st.keypress(None, None)
+                            if self.delegate.page_pile:
+                                nf = self.delegate.page_pile.focus
+                                if hasattr(nf, "key_timeout"):
+                                    st = nf
+                                elif hasattr(nf, "original_widget"):
+                                    no = nf.original_widget
+                                    if hasattr(no, "original_widget"):
+                                        st = no.original_widget
+                                    else:
+                                        if hasattr(no, "key_timeout"):
+                                            st = no
+                            
+                                if st and hasattr(st, "key_timeout") and hasattr(st, "keypress") and callable(st.keypress):
+                                    st.keypress(None, None)
 
                         nomadnet.NomadNetworkApp.get_shared_instance().ui.loop.set_alarm_in(0.25, df)
                 
@@ -111,6 +114,9 @@ class Browser:
         self.frame = None
         self.attr_maps = []
         self.page_pile = None
+        self.page_partials = {}
+        self.updater_running = False
+        self.partial_updater_lock = Lock()
         self.build_display()
 
         self.history = []
@@ -179,6 +185,7 @@ class Browser:
             return destination_type
 
     def handle_link(self, link_target, link_data = None):
+        partial_ids = None
         request_data = None
         if link_data != None:
             link_fields = []
@@ -229,12 +236,6 @@ class Browser:
                                 else:
                                     pass  # do nothing if checkbox is not check
 
-
-
-
-                        
-
-                
             recurse_down(self.attr_maps)
             RNS.log("Including request data: "+str(request_data), RNS.LOG_DEBUG)
 
@@ -244,6 +245,10 @@ class Browser:
         if len(components) == 2:
             destination_type = self.expand_shorthands(components[0])
             link_target = components[1]
+        elif link_target.startswith("p:"):
+            comps = link_target.split(":")
+            if len(comps) > 1: partial_ids = comps[1:]
+            destination_type = "partial"
         else:
             destination_type = "nomadnetwork.node"
             link_target = components[0]
@@ -264,6 +269,9 @@ class Browser:
             RNS.log("Passing LXMF link to handler", RNS.LOG_DEBUG)
             self.handle_lxmf_link(link_target)
 
+        elif destination_type == "partial":
+            if partial_ids != None and len(partial_ids) > 0: self.handle_partial_updates(partial_ids)
+
         else:
             RNS.log("No known handler for destination type "+str(destination_type), RNS.LOG_DEBUG)
             self.browser_footer = urwid.Text("Could not open link: "+"No known handler for destination type "+str(destination_type))
@@ -317,6 +325,7 @@ class Browser:
         self.browser_footer = urwid.Text("")
 
         self.page_pile = None
+        self.page_partials = {}
         self.browser_body = urwid.Filler(
             urwid.Text("Disconnected\n"+self.g["arrow_l"]+"  "+self.g["arrow_r"], align=urwid.CENTER),
             urwid.MIDDLE,
@@ -373,6 +382,7 @@ class Browser:
         if self.status == Browser.DISCONECTED:
             self.display_widget.set_attr_map({None: "inactive_text"})
             self.page_pile = None
+            self.page_partials = {}
             self.browser_body = urwid.Filler(
                 urwid.Text("Disconnected\n"+self.g["arrow_l"]+"  "+self.g["arrow_r"], align=urwid.CENTER),
                 urwid.MIDDLE,
@@ -446,7 +456,234 @@ class Browser:
         pile = urwid.Pile(self.attr_maps)
         pile.automove_cursor_on_scroll = True
         self.page_pile = pile
+        self.page_partials = {}
         self.browser_body = urwid.AttrMap(ScrollBar(Scrollable(pile, force_forward_keypress=True), thumb_char="\u2503", trough_char=" "), "scrollbar")
+        self.detect_partials()
+
+    def parse_url(self, url):
+        path = None
+        destination_hash = None
+        components = url.split(":")
+        if len(components) == 1:
+            if len(components[0]) == (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2:
+                try: destination_hash = bytes.fromhex(components[0])
+                except Exception as e: raise ValueError("Malformed URL")
+                path = Browser.DEFAULT_PATH
+            else: raise ValueError("Malformed URL")
+        elif len(components) == 2:
+            if len(components[0]) == (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2:
+                try: destination_hash = bytes.fromhex(components[0])
+                except Exception as e: raise ValueError("Malformed URL")
+                path = components[1]
+                if len(path) == 0: path = Browser.DEFAULT_PATH
+            else:
+                if len(components[0]) == 0:
+                    if self.destination_hash != None:
+                        destination_hash = self.destination_hash
+                        path = components[1]
+                        if len(path) == 0: path = Browser.DEFAULT_PATH
+                    else: raise ValueError("Malformed URL")
+                else: raise ValueError("Malformed URL")
+        else: raise ValueError("Malformed URL")
+
+        return destination_hash, path
+
+    def detect_partials(self):
+        for w in self.attr_maps:
+            o = w._original_widget
+            if hasattr(o, "partial_hash"):
+                RNS.log(f"Found partial: {o.partial_hash} / {o.partial_url} / {o.partial_refresh}")
+                partial = {"hash": o.partial_hash, "id": o.partial_id, "url": o.partial_url, "fields": o.partial_fields,
+                           "refresh": o.partial_refresh, "content": None, "updated": None, "update_requested": None, "request_id": None,
+                            "destination": None, "link": None, "pile": o, "attr_maps": None, "failed": False, "pr_throttle": 0}
+
+                self.page_partials[o.partial_hash] = partial
+
+        if len(self.page_partials) > 0: self.start_partial_updater()
+
+    def partial_failed(self, request_receipt):
+        RNS.log("Loading page partial failed", RNS.LOG_ERROR)
+        for pid in self.page_partials:
+            partial = self.page_partials[pid]
+            if partial["request_id"] == request_receipt.request_id:
+                try:
+                    partial["updated"] = time.time()
+                    partial["request_id"] = None
+                    partial["content"] = None
+                    partial["attr_maps"] = None
+                    url = partial["url"]
+                    pile = partial["pile"]
+                    pile.contents = [(urwid.Text(f"Could not load partial {url}: The resource transfer failed"), pile.options())]
+                except Exception as e:
+                    RNS.log(f"Error in partial failed callback: {e}", RNS.LOG_ERROR)
+                    RNS.trace_exception(e)
+
+    def partial_progressed(self, request_receipt):
+        pass
+
+    def partial_received(self, request_receipt):
+        for pid in self.page_partials:
+            partial = self.page_partials[pid]
+            if partial["request_id"] == request_receipt.request_id:
+                try:
+                    partial["updated"] = partial["update_requested"]
+                    partial["request_id"] = None
+                    partial["content"] = request_receipt.response.decode("utf-8").rstrip()
+                    partial["attr_maps"] = markup_to_attrmaps(strip_modifiers(partial["content"]), url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
+                    pile = partial["pile"]
+                    pile.contents = [(e, pile.options()) for e in partial["attr_maps"]]
+
+                except Exception as e:
+                    RNS.trace_exception(e)
+
+    def __load_partial(self, partial):
+        if partial["failed"] == True: return
+        try: partial_destination_hash, path = self.parse_url(partial["url"])
+        except Exception as e:
+            RNS.log(f"Could not parse partial URL: {e}", RNS.LOG_ERROR)
+            partial["failed"] = True
+            pile = partial["pile"]
+            url = partial["url"]
+            pile.contents = [(urwid.Text(f"Could not load partial {url}: {e}"), pile.options())]
+            return
+
+        if partial_destination_hash != self.loopback and not RNS.Transport.has_path(partial_destination_hash):
+            if time.time() <= partial["pr_throttle"]: return
+            else:
+                partial["pr_throttle"] = time.time()+15
+                RNS.log(f"Requesting path for partial: {partial_destination_hash} / {path}", RNS.LOG_EXTREME)
+                RNS.Transport.request_path(partial_destination_hash)
+                pr_time = time.time()+RNS.Transport.first_hop_timeout(partial_destination_hash)
+                while not RNS.Transport.has_path(partial_destination_hash):
+                    now = time.time()
+                    if now > pr_time+self.timeout: return
+                    time.sleep(0.25)
+
+        for pid in self.page_partials:
+            other_partial = self.page_partials[pid]
+            if other_partial["link"]:
+                existing_link = other_partial["link"]
+                if existing_link.destination.hash == partial_destination_hash and existing_link.status == RNS.Link.ACTIVE:
+                    RNS.log(f"Re-using existing link: {existing_link}", RNS.LOG_EXTREME)
+                    partial["link"] = existing_link
+                    break
+
+        if not partial["link"] or partial["link"].status == RNS.Link.CLOSED:
+            RNS.log(f"Establishing link for partial: {partial_destination_hash} / {path}", RNS.LOG_EXTREME)
+            identity = RNS.Identity.recall(partial_destination_hash)
+            destination = RNS.Destination(identity, RNS.Destination.OUT, RNS.Destination.SINGLE, self.app_name, self.aspects)
+            
+            def established(link):
+                RNS.log(f"Link established for partial: {partial_destination_hash} / {path}", RNS.LOG_EXTREME)
+
+            def closed(link):
+                RNS.log(f"Link closed for partial: {partial_destination_hash} / {path}", RNS.LOG_EXTREME)
+                partial["link"] = None
+            
+            partial["link"] = RNS.Link(destination, established_callback = established, closed_callback = closed)
+            timeout = time.time()+self.timeout
+            while partial["link"].status != RNS.Link.ACTIVE and time.time() < timeout: time.sleep(0.1)
+
+        if partial["link"] and partial["link"].status == RNS.Link.ACTIVE and partial["request_id"] == None:
+            RNS.log(f"Sending request for partial: {partial_destination_hash} / {path}", RNS.LOG_EXTREME)
+            receipt = partial["link"].request(path, data=self.__get_partial_request_data(partial), response_callback = self.partial_received,
+                                              failed_callback = self.partial_failed, progress_callback = self.partial_progressed)
+
+            if receipt: partial["request_id"] = receipt.request_id
+            else: RNS.log(f"Partial request failed", RNS.LOG_ERROR)
+
+    def __get_partial_request_data(self, partial):
+        request_data = None
+        if partial["fields"] != None:
+            link_data = partial["fields"]
+            link_fields = []
+            request_data = {}
+            all_fields = True if "*" in link_data else False
+
+            for e in link_data:
+                if "=" in e:
+                    c = e.split("=")
+                    if len(c) == 2:
+                        request_data["var_"+str(c[0])] = str(c[1])
+                else:
+                    link_fields.append(e)
+
+                def recurse_down(w):
+                    if isinstance(w, list):
+                        for t in w:
+                            recurse_down(t)
+                    elif isinstance(w, tuple):
+                        for t in w:
+                            recurse_down(t)
+                    elif hasattr(w, "contents"):
+                        recurse_down(w.contents)
+                    elif hasattr(w, "original_widget"):
+                        recurse_down(w.original_widget)
+                    elif hasattr(w, "_original_widget"):
+                        recurse_down(w._original_widget)
+                    else:
+                        if hasattr(w, "field_name") and (all_fields or w.field_name in link_fields):
+                            field_key = "field_" + w.field_name
+                            if isinstance(w, urwid.Edit):
+                                request_data[field_key] = w.edit_text
+                            elif isinstance(w, urwid.RadioButton):
+                                if w.state:
+                                    user_data = getattr(w, "field_value", None)  
+                                    if user_data is not None:
+                                        request_data[field_key] = user_data
+                            elif isinstance(w, urwid.CheckBox):
+                                user_data = getattr(w, "field_value", "1")
+                                if w.state:
+                                    existing_value = request_data.get(field_key, '')
+                                    if existing_value:
+                                        # Concatenate the new value with the existing one
+                                        request_data[field_key] = existing_value + ',' + user_data
+                                    else:
+                                        # Initialize the field with the current value
+                                        request_data[field_key] = user_data
+                                else:
+                                    pass  # do nothing if checkbox is not check
+
+            recurse_down(self.attr_maps)
+            RNS.log("Including request data: "+str(request_data), RNS.LOG_DEBUG)
+
+        return request_data
+
+    def start_partial_updater(self):
+        if not self.updater_running: self.update_partials()
+
+    def handle_partial_updates(self, partial_ids):
+        RNS.log(f"Update partials: {partial_ids}")
+        def job():
+            for pid in self.page_partials:
+                try:
+                    partial = self.page_partials[pid]
+                    if partial["id"] in partial_ids:
+                        partial["update_requested"] = time.time()
+                        self.__load_partial(partial)
+                except Exception as e: RNS.log(f"Error updating page partial: {e}", RNS.LOG_ERROR)
+
+        threading.Thread(target=job, daemon=True).start()
+
+    def update_partials(self, loop=None, user_data=None):
+        with self.partial_updater_lock:
+            def job():
+                for pid in self.page_partials:
+                    try:
+                        partial = self.page_partials[pid]
+                        if partial["failed"]: continue
+                        if not partial["updated"] or (partial["refresh"] != None and time.time() > partial["updated"]+partial["refresh"]):
+                            partial["update_requested"] = time.time()
+                            self.__load_partial(partial)
+                    except Exception as e: RNS.log(f"Error updating page partial: {e}", RNS.LOG_ERROR)
+
+            threading.Thread(target=job, daemon=True).start()
+            
+            if len(self.page_partials) > 0:
+                self.updater_running = True
+                self.app.ui.loop.set_alarm_in(1, self.update_partials)
+            else:
+                self.updater_running = False
 
     def identify(self):
         if self.link != None:
@@ -486,35 +723,25 @@ class Browser:
         components = url.split(":")
         if len(components) == 1:
             if len(components[0]) == (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2:
-                try:
-                    destination_hash = bytes.fromhex(components[0])
-                except Exception as e:
-                    raise ValueError("Malformed URL")
+                try: destination_hash = bytes.fromhex(components[0])
+                except Exception as e: raise ValueError("Malformed URL")
                 path = Browser.DEFAULT_PATH
-            else:
-                raise ValueError("Malformed URL")
+            else: raise ValueError("Malformed URL")
         elif len(components) == 2:
             if len(components[0]) == (RNS.Reticulum.TRUNCATED_HASHLENGTH//8)*2:
-                try:
-                    destination_hash = bytes.fromhex(components[0])
-                except Exception as e:
-                    raise ValueError("Malformed URL")
+                try: destination_hash = bytes.fromhex(components[0])
+                except Exception as e: raise ValueError("Malformed URL")
                 path = components[1]
-                if len(path) == 0:
-                    path = Browser.DEFAULT_PATH
+                if len(path) == 0: path = Browser.DEFAULT_PATH
             else:
                 if len(components[0]) == 0:
                     if self.destination_hash != None:
                         destination_hash = self.destination_hash
                         path = components[1]
-                        if len(path) == 0:
-                            path = Browser.DEFAULT_PATH
-                    else:
-                        raise ValueError("Malformed URL")
-                else:
-                        raise ValueError("Malformed URL")
-        else:
-            raise ValueError("Malformed URL")
+                        if len(path) == 0: path = Browser.DEFAULT_PATH
+                    else: raise ValueError("Malformed URL")
+                else: raise ValueError("Malformed URL")
+        else: raise ValueError("Malformed URL")
 
         if destination_hash != None and path != None:
             if path.startswith("/file/"):
@@ -813,7 +1040,7 @@ class Browser:
                     fg = self.markup[fgpos+5:endpos]
                     self.page_foreground_color = fg
 
-            self.attr_maps = markup_to_attrmaps(self.markup, url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
+            self.attr_maps = markup_to_attrmaps(strip_modifiers(self.markup), url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
             
             self.response_progress = 0
             self.response_speed = None
@@ -880,7 +1107,7 @@ class Browser:
                         fg = self.markup[fgpos+5:endpos]
                         self.page_foreground_color = fg
 
-                self.attr_maps = markup_to_attrmaps(self.markup, url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
+                self.attr_maps = markup_to_attrmaps(strip_modifiers(self.markup), url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
                 
                 self.response_progress = 0
                 self.response_speed = None
@@ -1032,7 +1259,7 @@ class Browser:
                     fg = self.markup[fgpos+5:endpos]
                     self.page_foreground_color = fg
 
-            self.attr_maps = markup_to_attrmaps(self.markup, url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
+            self.attr_maps = markup_to_attrmaps(strip_modifiers(self.markup), url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
             self.response_progress = 0
             self.response_speed = None
             self.progress_updated_at = None

nomadnet/ui/textui/Guide.py

@@ -1455,6 +1455,36 @@ This line will
 ``
 
 
+>Partials
+
+You can include partials in pages, which will load asynchronously once the page itself has loaded. 
+
+`Faaa
+`=
+`{f64a846313b874ee4a357040807f8c77:/page/partial_1.mu}
+`=
+``
+
+It's also possible to set an auto-refresh interval for partials. Omit or set to 0 to disable. The following partial will update every 10 seconds.
+
+`Faaa
+`=
+`{f64a846313b874ee4a357040807f8c77:/page/refreshing_partial.mu`10}
+`=
+``
+
+You can include field values and variables in partial updates, and by setting the `!pid`! variable, you can create links that update one or more specific partials.
+
+`Faaa
+`=
+Name: `B444`<user_name`>`b
+
+`F38a`[Say hello`p:32]`f
+
+`{f64a846313b874e84a357039807f8c77:/page/hello_partial.mu`0`pid=32|user_name}
+`=
+``
+
 >Literals
 
 To display literal content, for example source-code, or blocks of text that should not be interpreted by micron, you can use literal blocks, specified by the \\`= tag. Below is the source code of this entire document, presented as a literal block.

nomadnet/ui/textui/MicronParser.py

@@ -2,6 +2,7 @@ import nomadnet
 import urwid
 import random
 import time
+import RNS
 from urwid.util import is_mouse_press
 from urwid.text_layout import calc_coords
 
@@ -85,6 +86,54 @@ def markup_to_attrmaps(markup, url_delegate = None, fg_color=None, bg_color=None
 
     return attrmaps
 
+def parse_partial(line):
+    try:
+        endpos = line.find("}")
+        if endpos == -1: return None
+        else:
+            partial_data = line[0:endpos]
+
+            partial_id = None
+            partial_components = partial_data.split("`")
+            if len(partial_components) == 1:
+                partial_url = partial_components[0]
+                partial_refresh = None
+                partial_fields = ""
+            elif len(partial_components) == 2:
+                partial_url = partial_components[0]
+                partial_refresh = float(partial_components[1])
+                partial_fields = ""
+            elif len(partial_components) == 3:
+                partial_url = partial_components[0]
+                partial_refresh = float(partial_components[1])
+                partial_fields = partial_components[2]
+            else:
+                partial_url = ""
+                partial_fields = ""
+                partial_refresh = None
+
+            if partial_refresh != None and partial_refresh < 1: partial_refresh = None
+
+            pf = partial_fields.split("|")
+            if len(pf) > 0:
+                partial_fields = pf
+                for f in pf:
+                    if f.startswith("pid="):
+                        pcs = f.split("=")
+                        partial_id = pcs[1]
+
+            if len(partial_url):
+                pile = urwid.Pile([urwid.Text(f"⧖")])
+                partial_descriptor = "|".join(partial_components)
+                pile.partial_id = partial_id
+                pile.partial_hash = RNS.hexrep(RNS.Identity.full_hash(partial_descriptor.encode("utf-8")), delimit=False)
+                pile.partial_url = partial_url
+                pile.partial_fields = partial_fields
+                pile.partial_refresh = partial_refresh
+                return [pile]
+
+    except Exception as e: return None
+
 def parse_line(line, state, url_delegate):
     pre_escape = False
     if len(line) > 0:
@@ -106,6 +155,10 @@ def parse_line(line, state, url_delegate):
             elif first_char == "#":
                 return None
 
+            # Check for partials
+            elif line.startswith("`{"):
+                return parse_partial(line[2:])
+
             # Check for section heading reset
             elif first_char == "<":
                 state["depth"] = 0
@@ -283,14 +336,11 @@ def make_style(state):
 
                 if color[0] == "g":
                     val = int(color[1:2])
-                    if val < 25:
-                        result = "black"
-                    elif val < 50:
-                        result = "dark gray"
-                    elif val < 75:
-                        result = "light gray"
-                    else:
-                        result = "white"
+                    if val < 25:   result = "black"
+                    elif val < 50: result = "dark gray"
+                    elif val < 75: result = "light gray"
+                    else:          result = "white"
+                
                 else:
                     r = int(color[0], 16)
                     g = int(color[1], 16)
@@ -298,65 +348,43 @@ def make_style(state):
 
                     if r == g == b:
                         val = int(color[0], 16)*6
-                        if val < 12:
-                            result = "black"
-                        elif val < 50:
-                            result = "dark gray"
-                        elif val < 80:
-                            result = "light gray"
-                        else:
-                            result = "white"
+                        if val < 12:   result = "black"
+                        elif val < 50: result = "dark gray"
+                        elif val < 80: result = "light gray"
+                        else:          result = "white"
 
                     else:
                         if r == b:
                             if r > g:
-                                if r > t:
-                                    result = "light magenta"
-                                else:
-                                    result = "dark magenta"
+                                if r > t: result = "light magenta"
+                                else:     result = "dark magenta"
                             else:
-                                if g > t:
-                                    result = "light green"
-                                else:
-                                    result = "dark green"
+                                if g > t: result = "light green"
+                                else:     result = "dark green"
                         if b == g:
                             if b > r:
-                                if b > t:
-                                    result = "light cyan"
-                                else:
-                                    result = "dark cyan"
+                                if b > t: result = "light cyan"
+                                else:     result = "dark cyan"
                             else:
-                                if r > t:
-                                    result = "light red"
-                                else:
-                                    result = "dark red"
+                                if r > t: result = "light red"
+                                else:     result = "dark red"
                         if g == r:
                             if g > b:
-                                if g > t:
-                                    result = "yellow"
-                                else:
-                                    result = "brown"
+                                if g > t: result = "yellow"
+                                else:     result = "brown"
                             else:
-                                if b > t:
-                                    result = "light blue"
-                                else:
-                                    result = "dark blue"
+                                if b > t: result = "light blue"
+                                else:     result = "dark blue"
 
                         if r > g and r > b:
-                            if r > t:
-                                result = "light red"
-                            else:
-                                result = "dark red"
+                            if r > t: result = "light red"
+                            else:     result = "dark red"
                         if g > r and g > b:
-                            if g > t:
-                                result = "light green"
-                            else:
-                                result = "dark green"
+                            if g > t: result = "light green"
+                            else:     result = "dark green"
                         if b > g and b > r:
-                            if b > t:
-                                result = "light blue"
-                            else:
-                                result = "dark blue"
+                            if b > t: result = "light blue"
+                            else:     result = "dark blue"
 
         except Exception as e:
             result = "default"
@@ -422,12 +450,9 @@ def make_style(state):
     bg        = state["bg_color"]
 
     format_string = ""
-    if bold:
-        format_string += ",bold"
-    if underline:
-        format_string += ",underline"
-    if italic:
-        format_string += ",italics"
+    if bold:      format_string += ",bold"
+    if underline: format_string += ",underline"
+    if italic:    format_string += ",italics"
 
     name = "micron_"+fg+"_"+bg+"_"+format_string
     if not name in SYNTH_STYLES:
@@ -488,13 +513,10 @@ def make_output(state, line, url_delegate, pre_escape=False):
                         state["align"] = state["default_align"]
                     elif c == "c":
                         if state["align"] != "center": state["align"] = "center"
-                        # else: state["align"] = state["default_align"]
                     elif c == "l":
                         if state["align"] != "left": state["align"] = "left"
-                        # else: state["align"] = state["default_align"]
                     elif c == "r":
                         if state["align"] != "right": state["align"] = "right"
-                        # else: state["align"] = state["default_align"]
                     elif c == "a":
                         state["align"] = state["default_align"]
 
@@ -651,9 +673,7 @@ def make_output(state, line, url_delegate, pre_escape=False):
 
                                     output.append((linkspec, link_label))
                                 else:
-                                    output.append(make_part(state, link_label)) 
-
-                                
+                                    output.append(make_part(state, link_label))
 
                     mode = "text"
                     if len(part) > 0:

nomadnet/ui/textui/Network.py

@@ -6,6 +6,7 @@ import threading
 from datetime import datetime
 from nomadnet.Directory import DirectoryEntry
 from nomadnet.vendor.additional_urwid_widgets import IndicativeListBox, MODIFIER_KEY
+from nomadnet.util import strip_modifiers
 
 from .Browser import Browser
 
@@ -84,7 +85,7 @@ class AnnounceInfo(urwid.WidgetWrap):
             type_string = "Peer " + g["peer"]
 
         try:
-            data_str = announce[2].decode("utf-8")
+            data_str = strip_modifiers(announce[2].decode("utf-8"))
             data_style = ""
             if trust_level != DirectoryEntry.TRUSTED and len(data_str) > 32:
                 data_str = data_str[:32]+" [...]"
@@ -250,7 +251,7 @@ class AnnounceInfo(urwid.WidgetWrap):
 
 
 class AnnounceStreamEntry(urwid.WidgetWrap):
-    def __init__(self, app, announce, delegate):
+    def __init__(self, app, announce, delegate, show_destination=False):
         full_time_format = "%Y-%m-%d %H:%M:%S"
         date_time_format = "%Y-%m-%d"
         time_time_format = "%H:%M:%S"
@@ -274,7 +275,16 @@ class AnnounceStreamEntry(urwid.WidgetWrap):
             ts_string = dt.strftime(date_only_format)
 
         trust_level  = self.app.directory.trust_level(source_hash)
-        display_str = self.app.directory.simplest_display_str(source_hash)
+        
+        if show_destination:
+            display_str = RNS.hexrep(source_hash, delimit=False)
+        else:
+            try:
+                display_str = strip_modifiers(announce[2].decode("utf-8"))
+                if len(display_str) > 32:
+                    display_str = display_str[:32] + "..."
+            except:
+                display_str = self.app.directory.simplest_display_str(source_hash)
 
         if trust_level == DirectoryEntry.UNTRUSTED:
             symbol        = g["cross"]
@@ -381,22 +391,33 @@ class AnnounceStream(urwid.WidgetWrap):
         self.ilb = None
         self.no_content = True
         self.current_tab = "nodes"
+        self.show_destination = False
+        self.search_text = ""
 
         self.added_entries = []
         self.widget_list = []
-        self.update_widget_list()
 
-        # Create tab buttons
-        self.tab_nodes = TabButton("Nodes", on_press=self.show_nodes_tab)
-        self.tab_peers = TabButton("Peers", on_press=self.show_peers_tab)
-        self.tab_pn = TabButton("Propagation Nodes", on_press=self.show_pn_tab)
+        self.tab_nodes = TabButton("Nodes (0)", on_press=self.show_nodes_tab)
+        self.tab_peers = TabButton("Peers (0)", on_press=self.show_peers_tab)
+        self.tab_pn = TabButton("Propagation Nodes (0)", on_press=self.show_pn_tab)
 
-        # Create tab bar with proportional widths
         self.tab_bar = urwid.Columns([
             ('weight', 1, self.tab_nodes),
             ('weight', 1, self.tab_peers),
             ('weight', 3, self.tab_pn),
-        ], dividechars=1)  # Add 1 character spacing between tabs
+        ], dividechars=1)
+
+        self.search_edit = urwid.Edit(caption="Search: ")
+        urwid.connect_signal(self.search_edit, 'change', self.on_search_change)
+
+        self.display_toggle = TabButton("Show: Name", on_press=self.toggle_display_mode)
+
+        self.filter_bar = urwid.Columns([
+            ('weight', 2, self.search_edit),
+            ('weight', 1, self.display_toggle),
+        ], dividechars=1)
+
+        self.update_widget_list()
 
         self.ilb = ExceptionHandlingListBox(
             self.widget_list,
@@ -406,9 +427,9 @@ class AnnounceStream(urwid.WidgetWrap):
             #highlight_offFocus="list_off_focus"
         )
 
-        # Combine tab bar and list box
         self.pile = urwid.Pile([
             ('pack', self.tab_bar),
+            ('pack', self.filter_bar),
             ('weight', 1, self.ilb),
         ])
 
@@ -416,13 +437,25 @@ class AnnounceStream(urwid.WidgetWrap):
         super().__init__(urwid.LineBox(self.display_widget, title="Announce Stream"))
 
     def keypress(self, size, key):
-        if key == "up" and (self.no_content or self.ilb.first_item_is_selected()):
+        if key == "up" and self.pile.focus == self.tab_bar:
             nomadnet.NomadNetworkApp.get_shared_instance().ui.main_display.frame.focus_position = "header"
         elif key == "ctrl x":
             self.delete_selected_entry()
 
         return super(AnnounceStream, self).keypress(size, key)
 
+    def on_search_change(self, widget, text):
+        self.search_text = text.lower()
+        self.update_widget_list()
+
+    def toggle_display_mode(self, button):
+        self.show_destination = not self.show_destination
+        if self.show_destination:
+            self.display_toggle.set_label("Show: Dest")
+        else:
+            self.display_toggle.set_label("Show: Name")
+        self.update_widget_list()
+
     def delete_selected_entry(self):
         if self.ilb.get_selected_item() != None:
             self.app.directory.remove_announce_with_timestamp(self.ilb.get_selected_item().original_widget.timestamp)
@@ -438,19 +471,36 @@ class AnnounceStream(urwid.WidgetWrap):
         self.widget_list = []
         new_entries = []
 
+        node_count = 0
+        peer_count = 0
+        pn_count = 0
+
         for e in self.app.directory.announce_stream:
             announce_type = e[3]
 
-            # Filter based on current tab
-            if self.current_tab == "nodes" and (announce_type == "node" or announce_type == True):
-                new_entries.append(e)
-            elif self.current_tab == "peers" and (announce_type == "peer" or announce_type == False):
-                new_entries.append(e)
-            elif self.current_tab == "pn" and announce_type == "pn":
-                new_entries.append(e)
+            if self.search_text:
+                try:
+                    announce_data = e[2].decode("utf-8").lower()
+                except:
+                    announce_data = ""
+                if self.search_text not in announce_data:
+                    continue
+
+            if announce_type == "node" or announce_type == True:
+                node_count += 1
+                if self.current_tab == "nodes":
+                    new_entries.append(e)
+            elif announce_type == "peer" or announce_type == False:
+                peer_count += 1
+                if self.current_tab == "peers":
+                    new_entries.append(e)
+            elif announce_type == "pn":
+                pn_count += 1
+                if self.current_tab == "pn":
+                    new_entries.append(e)
 
         for e in new_entries:
-            nw = AnnounceStreamEntry(self.app, e, self)
+            nw = AnnounceStreamEntry(self.app, e, self, show_destination=self.show_destination)
             nw.timestamp = e[0]
             self.widget_list.append(nw)
 
@@ -460,6 +510,10 @@ class AnnounceStream(urwid.WidgetWrap):
             self.no_content = True
             self.widget_list = [urwid.Text(f"No {self.current_tab} announces", align='center')]
 
+        self.tab_nodes.set_label(f"Nodes ({node_count})")
+        self.tab_peers.set_label(f"Peers ({peer_count})")
+        self.tab_pn.set_label(f"Propagation Nodes ({pn_count})")
+
         if self.ilb:
             self.ilb.set_body(self.widget_list)
 
@@ -555,7 +609,7 @@ class KnownNodeInfo(urwid.WidgetWrap):
         if node_entry == None:
             display_str = self.app.directory.simplest_display_str(source_hash)
         else:
-            display_str = node_entry.display_name
+            display_str = strip_modifiers(node_entry.display_name)
 
         addr_str     = "<"+RNS.hexrep(source_hash, delimit=False)+">"
 
@@ -1648,7 +1702,6 @@ class NetworkDisplay():
     def reinit_known_nodes(self):
         self.known_nodes_display = KnownNodes(self.app)
         self.known_nodes_display.delegate = self
-        self.close_list_dialogs()
         self.announce_stream_display.rebuild_widget_list()
 
     def reinit_lxmf_peers(self):
@@ -1908,4 +1961,4 @@ def pretty_date(time=False):
         return str(int(day_diff / 7)) + " weeks ago"
     if day_diff < 365:
         return str(int(day_diff / 30)) + " months ago"
-    return str(int(day_diff / 365)) + " years ago"
+    return str(int(day_diff / 365)) + " years ago"

nomadnet/util.py

@@ -0,0 +1,35 @@
+import re
+import unicodedata
+
+invalid_rendering = ["🕵️", "☝"]
+
+def strip_modifiers(text):
+    def process_characters(text):
+        result = []
+        i = 0
+        while i < len(text):
+            char = text[i]
+            category = unicodedata.category(char)
+            
+            if category.startswith(('L', 'N', 'P', 'S')):
+                result.append(char)
+                i += 1
+            elif category.startswith(('M', 'Sk', 'Cf')) or char in '\u200d\u200c':
+                i += 1
+            else:
+                result.append(char)
+                i += 1
+                
+        return ''.join(result)
+
+    for char in invalid_rendering:
+        text = text.replace(char, " ")
+    
+    stripped = process_characters(text)
+    stripped = re.sub(r'[\uFE00-\uFE0F]', '', stripped)
+    stripped = re.sub(r'[\U000E0100-\U000E01EF]', '', stripped, flags=re.UNICODE)
+    stripped = re.sub(r'[\U0001F3FB-\U0001F3FF]', '', stripped, flags=re.UNICODE)
+    stripped = re.sub(r'[\u200D\u200C]', '', stripped)
+    stripped = re.sub(r'\r\n?', '\n', stripped)
+    
+    return stripped