You are here

Node memory limitations

13 posts / 0 new
Last post
kc8fyi
Node memory limitations
I am new to the mesh world but thanks to the help of individuals and forum posts here I am making a go of it as a winter project. While researching what applications I might want to try, mesh chat specifically at this time, I came across one post that got me thinking. The post was while discussing mesh chat installation on a node and the person said they wouldn't suggest loading anything like mesh chat on a node as it would take up memory and degrade service overall. I'm sorry I can't find that post for reference and haven't found any others that say similar things. I know you probably could load enough onto a node to effect it but is that an issue so long as you're realistic? Are there certain apps that aren't recommended due to heavy resource demands? Thanks in advance for any info. Neil
AA7AU
AA7AU's picture
Personal Opnion

I know it has been written here several times by very knowledgeable folks that MeshChat uses too much memory and resource when installed on a node itself and therefore should rather be installed on an attached Raspberry Pi. And, it is true that PI does offer the benefit of the much greater "files" storage space under that app, which enables nice file transfer/backup/sharing/coordination etc.

However I respectfully suggest that MeshChat can work extremely well when installed directly on a node (running the stable firmware version) when the installer understands the trade-offs and proper procedure. One important part is to install the tunnel first (if you are ever going to use one) before installing MeshChat. Also, it helps to disable it when doing a firmware upgrade, Also, pick a unique zone name for your chat (and then power down your node for 15 minutes to start/reset the DNS advertisments. with your new unique zone name).

So, for example. here's some real life experience - I have four (sometimes five) instances named as a [shared] single mesh chat zone installed in the LV metro area, and each one is installed on the node itself. This is working extremely well. Most of those nodes are basically "bare" nodes with nothing much attached or dragging down backup power etc. I also have a similar setup in central Idaho, but up there I use two separate zones (on different sets of nodes), one for EMS/medical (hospital based) and the other for general CNC comms. Same thing, MeshChat always lives in the node itself, even though we do use several Raspberry PIs for other mesh application (PBX, web, remote-HF, etc).

When they are all on the same mesh with the same "zone name", MeshChat automatically synchronizes the "database" of posts between all of those instances (doesn't work over tunnels) so that chat posting database will survive node shutdowns unless all five are down at the exact same time. Otherwise it replicates and repopulates - multiple instances are therefore normally self-healing and self-propagating! (Typos never die!).

One other side benefit of multiple instances is the one can connect/use the "closest" instance for more operator real-time response when a more distant node might have a less responsive connection/latency/etc.

I understand that the developers have informed us all that they may run out of "space" as AREDN grows/matures (who knows what Openwrt will do with RAM). If that time ever comes, I'll be faced with the choice to re-engineer my setups or simply freeze them at the last workable firmware version.

In the meantime. MeshChat may not be super-complex or sophisticated but, in its simplicity, it sure works well for us.

Again: this is all offered as personal opinion, not as advice. YMMV. I look forward to more posts here regarding "Node memory limitations".

HTH,
- Don - AA7AU

kc8fyi
Thanks for the quick and
Thanks for the quick and complete reply. It was good to hear of a real life example as well. Though it was a bit disheartening that this could even be an issue since mesh chat seems built specific for this application. In my limited initial setup I shouldn't have any issues being so small, but I would hate to invest time into setup and learning the system, on the node or a pi, if I may outgrow it. I have been perusing the forums for other options people like that are nearly as resilient. I have a lot to learn. Keep the info and (constructive) opinions coming please.
AE6XE
AE6XE's picture
The risk area is on devices
The risk area is on devices with 32Mb RAM with known issues if co-installing with tunnels or other packages.    All of the meshchat messages are cached in RAM consuming space, be sure to disable before any firmware upgrades.   The image sizes are 6Mb and growing, which needs space available in RAM during the upgrade process.

Joe AE6XE
kc8fyi
Thanks Joe for the heads up
Thanks Joe for the heads up and for past help via email. I'll look into disabling it before updates and maybe limiting the amount of past messages it caches. I'm starting to think a few strategically placed pies running the needed apps might be better and offer redundancy. I will say that in the official docs they never, that I saw, mention how the apps might affect performance. That's why I started to look into meshchat. Maybe a future edit of those could mention performance even briefly. The search for info continues.
K5DLQ
K5DLQ's picture
https://arednmesh.readthedocs
kc8fyi
I figured someone would point
I figured someone would point out something to prove me wrong. I don't know how I missed that. My mistake.
kc8fyi
   Over the last couple days
   Over the last couple days I have been re-reading through sections of the "how to" docs and have noticed several instances where it mentions not loading up the nodes.  It sure smells like breakfast with all this egg on my face.  Anyhow...
   The question that came to mind was exactly how much memory does the operating system take as a percentage, but I know that is dependent on which device.  So I guess a question that might be easier to answer is since its inception how much has the AREDN image grown?  Also, for a bonus question, what language is the image written in?  I have done some programming over the years even though it isn't my profession, but have never looked into hardware coding.

Neil
 
AE6XE
AE6XE's picture
A stripped down bare-bone
A stripped down bare-bone linux kernel to boot these devices will consume just over 4MB of flash space now days.   The RAM used to run the kernel would take up more than 16MB.     We haven't talked about a User Interface to configure the device.  Also, how to do routing automatically, how to monitor the device keeps running as expected, and more.   Today the minimum is 8MB of Flash with 32MB RAM and we have seen issues on these devices.  This is a combination of kernel bugs (memory leaks not freeing up RAM), and loading additional capabilities (meshchat, tunnels, iperf, ...) 

10 years ago, a 4MB Flash with 16MB RAM device would work, but then it didn't do everything we have grown to expect as standard today.   

Today, the images sizes are passing 6MB.  This has to fit in flash memory, then we have to reserve space for saving settings and other persistent information to run the device =  ~6MB + ~1MB = ~7MB now.   Then, we want to add optional capability (tunnels, etc.).   All of these can add up to over 8MB, with no space to install them.   
 
The RAM has to have 6MB+ of available space to upload the next biggest-greatest firmware to remote ';sysupgrade', unless we accept to drive up to tower sites to use TFTP processes and overwrite all our settings.  (Careful , the free memory value on mesh status page is a guess and not exactly that accurate.  The linux kernel itself doesn't actually have an exact number.)    The current 8/32 Flash/RAM devices are still working.   This next release has fixed some kernel memory leak issues that we hope will make them more stable, thus extending their life for many more years.  The jury is still out. 

However, 16/64 (or bigger) devices are same cost now days, and should be the 'go-to' option for anyone buying new devices.    The exception is the Nanostation or Nanobridge 3GHz devices, where there aren't other options. 

Joe AE6XE
 
AB7PA
Size of the firmware image is only one factor

This is just my anecdotal observation....   The size of the firmware image is fairly consistent across different vendor models, but the amount of free memory varies widely based on the type of device on which it is installed.  For example, compare the free memory stats on a 64 mb Rocket to the comparably-priced BaseBox with 64 mb memory and the same firmware release:

  • Rocket: ~1,200 kb free flash and ~12,000 kb free memory
  • BaseBox: ~9,000 kb free flash and ~34,000 kb free memory

One of our ​Mikrotik QRT5 devices has free flash ~98,000 kb and free memory ~34,000 kb.  Different devices can have different free memory footprints, even with the same firmware release installed.

AE6XE
AE6XE's picture
Generally, the free memory

Generally, the free memory value on the mesh status page needs to be interpreted with caution.  This value should be adjusted by subtracting the amount of /tmp consumed (something I want to put in the code), to become a little more accurate.   (meshchat stores messages in /tmp.) 
 
The code creating the images for all these devices has a great deal of commonality.

The QRT5 has a lot of flash at 128MB nand.   With only ~6MB images, lots of space.

I'd expect the Rocket to show a lot more free, the number doesn't add up.  What's the uptime on it?    

Joe

kc8fyi
   That makes sense about the
   That makes sense about the images not being too different in size.  Unless there was some momentous change in hardware the code needed to overcome it should be relatively similar.  I am curious about Joes last question?  What is the uptime?  I understand that most every computer slowly eats up space as it continues to run and this can be exacerbated by adding memory intensive programs like Meshchat into the mix.  What, if any, is the recommended time between power cycles to clear out any memory build up in a node without any extra apps loaded?
 
AE6XE
AE6XE's picture
The available RAM should not
The available RAM should not be decreasing over time unless there are defects.   In our case, these are the defects fixed in the upcoming upgrade below.   A  'memory leak' was reproduced by repeatedly running a program "iwinfo".    The fixes below were created to resolve this memory leak.   AREDN nodes run this command a couple times every minute, to capture the SNR history and more.    I would expect even 64MB RAM device to run out of memory after some (probably long) period of time.  

https://github.com/openwrt/openwrt/commit/466d499d033946a6e6799365f15e98680ca27b08

rpcd package update:
d442d62 plugin: fix double free in finish callback
ee26d83 main: exec_self: make clang analyzer happy
90e40bd file: exec: properly free memory on error
9ecfada uci: free configs list memory on return
32fba36 exec: always call finish_cb to allow plugin to free up memory
ca3e2d5 plugin: do not free method name separately
02c6e1d exec: properly free memory on rpc_exec() error
cc50263 plugin: exec: properly free memory on parse error
bd0ed25 uci: reset uci_ptr flags when merging set operations
37aa919 plugin: fix leaking invoked method name for exec plugins

 

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer