High memory consumption with multiple reports

General
1

What is deployed on: Everything is launched via docker compose.

The ui container and sheduler use syslog, the bunkerweb container uses the docker json driver with rotation.

The redis config is like this:

maxmemory 4gb
maxmemory-policy allkeys-lru
save “”
appendonly yes
appendfilename “appendonly.aof”
appendfsync everysec
auto-aof-rewrite-percentage 50
auto-aof-rewrite-min-size 2gb
timeout 0
tcp-keepalive 60
maxclients 10000
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes
no-appendfsync-on-rewrite yes
logfile “/data/redis.log”
dir /data

It is also used by redis and vector (for generating metrics) . I have a relatively average workload for projects. 24,568 incident reports, total requests 5.7M, blocked requests
10.8k have accumulated in 2 days. If you do not go into the UI, the state of the containers is as follows:

image
image822×173 7.34 KB

As soon as I try to log into the web ui, the ui container starts eating up memory (while logging into the ui for a very long time): and I also disabled logging via syslog from the bunkerweb container, otherwise it ate 40GB in 2 days!!!

image
image995×178 8.78 KB

The same situation occurs if I switch pages with reports, each time I have to wait up to almost a minute to go to the next page, and memory is jumping all the time.

If I understand the logic of the work correctly: all reports are written only in redis, it puts them in aof just in case, and when you log in to the ui, this case starts loading first back into redis, then everything into the ui, and therefore the ui consumes 8 - 8 GB of memory and as soon as it digests everything, it releases memory and the input occurs.Is there any way to fix this? Why, for example, can’t redis add reports that are older than 24 hours to a shared database, for example? This should not affect the efficiency of logging in any way.

I wrote the text through a translator, I’m sorry)

2

Hi @sheffnik,

ModSecurity is known to have issues with high RAM consumption and even memory leaks, particularly in configurations involving concurrent logging or when handling multiple rules, vhosts, or heavy workloads. Reports and bug records confirm that RAM usage can climb steadily and remain high until logs are manually cleared or services are restarted, which is a common pain point even with default or recommended ModSecurity configurations.​

The recent BunkerWeb release 1.6.6 includes targeted optimizations for reducing RAM consumption of individual services. It is recommended to update to this version and follow the official guide on optimizing RAM usage, especially when deploying many services in production environments. For details, see the BunkerWeb documentation for advanced optimization steps: Advanced usages - BunkerWeb documentation

References

If you are experiencing high memory consumption, upgrading your stack and following these best practices will help stabilize and reduce unneeded resource use.

Regards,

Théophile.

3

Thank you. Then I’ll look at MoD Sec in more detail. My version is 1.6.6 latest. It’s gotten a little better, but he’s still eating… Another option with syslog consumes a lot of resources (2TB passed there in a couple of days) may indicate to you in the documentation that using syslog on highly loaded services is not very good. It seems that even docker itself said this. But I could be wrong)

db config

image: mariadb:11
container_name: bw-db
command:

  • –max-allowed-packet=67108864
  • –max-connections=500
  • –wait-timeout=600
4

Do I understand correctly: I’m doing a global config with these exception settings?:

However, this limitation can be bypassed by writing global modsec-crs exclusion rules like this:

SecRule REQUEST_FILENAME "@rx ^/somewhere$" "nolog,phase:4,allow,id:1010,chain"
SecRule REQUEST_HEADERS:Host "@rx ^app1\.example\.com$" "nolog"

You can enable the global CRS loading by setting USE_MODSECURITY_GLOBAL_CRS to yes.

And then my custom configs for each service will work as before without enabling the global config?

Is this how the global config is created?

image
image876×235 7.95 KB

5

Hi @sheffnik,

Indeed syslog missed a logrotation config, we’ve recently updated our documentation about it.

I’d recommend using the latest syslog-ng version as the logrotation has been implemented in the 4.10.0 version.

Here’s how the new configuration example file looks:

@version: 4.10

# Source configuration to receive logs from Docker containers
source s_net {
  udp(
    ip("0.0.0.0")
  );
};

# Template to format log messages
template t_imp {
  template("$MSG\n");
  template_escape(no);
};

# Destination configuration to write logs to dynamically named files
destination d_dyna_file {
  file(
    "/var/log/bunkerweb/${PROGRAM}.log"
    template(t_imp)
    owner("101")
    group("101")
    dir_owner("root")
    dir_group("101")
    perm(0440)
    dir_perm(0770)
    create_dirs(yes)
    logrotate(
      enable(yes),
      size(100MB),
      rotations(7)
    )
  );
};

# Log path to direct logs to dynamically named files
log {
  source(s_net);
  destination(d_dyna_file);
};
6

Yes this doesn’t change the behavior of custom configurations. The “only” thing that changes is that the OWASP CoreRuleSet is loaded only once instead of loading it for each service.

7

At the moment, I have been using system rotation (logs are stored in volumes in a folder and are already rotated there via logorate) But I like the rotation option from syslog itself better, thank you).But again, the question is about the overall load on the system when using syslog - is it sure it won’t become a bottleneck under heavy loads?

8

If you configure the logrotation correctly then I’d say no.

1 Like
9

I have a question here. But will the scheme work with one common config, but with different settings for domains if I have a part on *.example.com (about 100 subdomains 1.example.com , 2.example.com ) but at the same time there are separate configs for several api.example.com api2.example.com . I won’t have to make separate links to all 100 subdomains in the case of *.example.com ?

10

Hi @sheffnik,

You can do so using templates as the SERVER_NAME setting doesn’t support wildcard.

Regards,

Théophile