Logs
The four places Modbot writes its log - the console, log files, Seq, and its own database, which is what the Modbot's log page in the app reads.
Modbot writes one log, to as many as four places at once:
| Where | Needs | Good for |
|---|---|---|
| The console | nothing | docker logs, and your hosting platform's log view |
| Log files | a disk that survives a restart | reading a whole run, and searching with a tool |
| Seq | SEQ_URL | searching everything, including the times Modbot cannot reach its database |
| The database | nothing | the Modbot's log page in the app |
The first three are described under Checking a deployment. This page is about the fourth.
The Modbot's log page
Modbot's log is in the sidebar under System. You need the See the operational log permission.
It exists for the deployment that has none of the other three to hand: a container whose disk is thrown away on every redeploy, no Seq server, and a hosting dashboard that shows the last few hundred lines with no way to search them.
Each row is one line: the time, the level, the message and where it came from. The level is coloured, and a warning, an error or a fatal line is marked down its left edge so it stands out while you read down the page.
Open a row to see the full message, the whole line as JSON — the time, the level, the message, the template, the source, the area, the properties it carried and the exception — and the exception on its own underneath, where a stack trace is readable. The copy button on the JSON copies the whole line.
The filters are:
| Filter | What it does |
|---|---|
| Search | Matches the message and the exception, anywhere in either. |
| Level | The lowest level shown. Warning gives warnings, errors and fatal lines. |
| Source | The part of Modbot that wrote it. |
| Area | Sync, Moderation, Analytics, Setup or Discord. |
| From and To | Whole days. |
What is in the table, and what is not
The table holds whatever level Modbot is recording at, which is Information and above unless you asked for more.
Requests Modbot sends to VRChat and Discord are not in it. They are in
modbot_log_http_<date>_<time>.jsonl and in Seq. A busy sync writes tens of thousands of those a
day, and six months of them would be larger than everything else Modbot stores put together.
Seeing every request and every query
Two kinds of line are written at Debug, so they are out of the way until you ask for them:
- one line per request Modbot answered — the method, the path, the status and how long it took;
- one line per database query Modbot ran.
Set LOG_LEVEL=Debug and both appear everywhere, the Modbot's
log page included. Set it back and they stop. A request that failed or answered a 5xx is a warning
whatever the level, so you never have to turn anything on to see the ones that went wrong.
MODBOT_DEBUG_LOGGING does the same for the Debug log files, and reaches the Modbot's log page as well.
Setting the level to Debug fills the table quickly. A quiet deployment writes a few thousand lines a day at Information and two or three hundred thousand a day at Debug, so the ceiling below is reached in about a week and the Modbot's log page then holds the last week rather than the last six months. Nothing breaks, and putting the level back stops it. What is already stored is still sent to Modbot Cloud, so leaving the level down sends your request paths and your SQL there too.
Secrets never reach the table. A property whose name looks like a credential — anything with
password, secret, token, apikey, authorization, cookie or credential in it — is stored
as [removed], and a bearer token or a connection-string password written into a message is taken
out of the text as well.
How long lines are kept
Settings, Server, Keep for (days), Modbot's log. 180 days by default, and 0
keeps them forever. Changing it needs the Change settings permission.
Once a day, Modbot deletes the lines past the window, ten thousand rows at a time.
There is a second limit you cannot change: two million lines, about a gigabyte. The same daily
job deletes the oldest past that and writes a warning saying it did. Keep-for is about time and this
one is about disk, so it applies even when keep-for is 0 — a Modbot that has filled its own
database takes your group's history down with it, and losing the oldest log lines does not.
The setting covers the database copy only. Log files have their own limit on how many are kept (see Checking a deployment), and Seq has its own retention.
Sending logs to Modbot Cloud
Modbot sends the same lines on to Modbot Cloud. It is on unless you
turn it off, in Settings, Server, Keep for (days), Send logs to Modbot Cloud, and
it is off entirely when MODBOT_CLOUD_DISABLED is set — that variable wins over the switch.
Why it is there: a deployment the project cannot reach is a deployment nobody can help with. You write "it stopped syncing yesterday", and the one thing that would answer it was in a container that has since been redeployed. Cloud holds a copy so the answer survives, and so you can read it from somewhere your own Modbot is not.
What is sent is exactly what is in your own table, unchanged: the same lines, the same properties, the same secrets already taken out. Nothing is filtered to warnings only and nothing is anonymised. Lines can name people: Modbot writes about its own work and that work is about members, so a line can carry a VRChat user id and may carry a name or an action taken on someone. Only credentials are taken out. See Privacy.
Your Modbot sends as the server it already registered as, the same one it reports with. That is what lets you read your own logs in Modbot Cloud once you have claimed the server: sign in there, pick your server, and the same filters are waiting. A Modbot Cloud administrator can read them too, which is the point of the copy — so somebody can help with a problem on a deployment they cannot reach.
A Modbot that has not finished registering with Cloud yet sends nothing and says so on Sync health. Registering happens on its own within a few minutes of starting.
The usage report makes that registration. Send usage report to Modbot Cloud in Settings, Server, This install is a separate switch from Send logs to Modbot Cloud, and it is on when Modbot is installed. With it off, a Modbot that never registered sends no logs, and Get link code is the one thing that registers it (sending only its address, version and operating system). If Cloud forgets a server that had registered, nothing registers it again while the usage report is off, so the logs stop until you switch it back on and it has registered again.
When Cloud cannot be reached
Nothing stops and nothing is queued twice. Modbot remembers the last line it sent and carries on from there, so a Cloud that is down for an afternoon costs nothing but freshness. It tries again on a backoff that grows from thirty seconds to ten minutes.
If Cloud stays unreachable long enough for 50,000 lines to pile up — about a fortnight for a quiet deployment — Modbot gives up on the oldest ones and counts them. The newest are the ones somebody is asking about. A batch Cloud refuses outright is skipped for the same reason: retrying it would never work and would block every line behind it.
Health says whether sending is on, when the last batch went, how many lines are waiting and how many were given up on.
What it costs
A line is about half a kilobyte stored. A quiet deployment writes a few thousand a day, so six months is a few hundred megabytes at most — the same order as a month of presence facts. If a deployment is writing far more than that, the log lines themselves are the thing to look at.
When the database is the problem
The Modbot's log page is read from the database, so it cannot tell you why the database is unreachable. That
is what SEQ_URL is for, and why it is an environment variable rather than a setting. See
Environment variables.
Lines written while the database is unreachable are held in memory — ten thousand of them — and written when it comes back. Past that, the oldest are dropped and counted, and the count is on the Health page.