Commit Graph

67 Commits

Author SHA1 Message Date
Michele Di Maria
b8fd88fba9 Fix the saving of the statistics on PosgreSQL #14124 (#14129)
* fix: deduplicate postgres sorted set bulk ops to prevent pkey violation

sortedSetIncrByBulk and sortedSetAddBulk did not deduplicate (key, value)
pairs before INSERT, causing "duplicate key value violates unique constraint
legacy_zset_pkey" errors since PostgreSQL ON CONFLICT only resolves against
existing table rows, not within-statement duplicates.

Also adds missing pageviews:ap metrics to analyticsKeys sorted set.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use upsert with RETURNING to prevent postgres analytics write failures

Replace the INSERT ON CONFLICT DO NOTHING + separate SELECT verification
pattern with INSERT ON CONFLICT DO UPDATE RETURNING. The old pattern had
an unreliable gap between INSERT and SELECT causing random "failed to
insert keys for objects" errors that blocked all analytics writes.

The no-op upsert (DO UPDATE SET type = existing type) guarantees every
row is returned via RETURNING, eliminating the need for a separate SELECT
and the "missing keys" check entirely. Also deduplicates the keys array
to prevent "cannot affect row a second time" errors with DO UPDATE.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 13:24:34 -04:00
Barış Soner Uşaklı
6c4e928482 fix: on exit, dont write analytics data on all nodes
if you are running 4 nodebbs each one was calling writeData which could trigger duplicate key errors
2026-03-27 16:45:20 -04:00
Barış Uşaklı
3c0a654012 feat: show cronjobs in acp (#14068)
* feat: show cronjobs in acp

add a wrapper for cronjobs and display in acp

* test: add running to spec

* test: fix running, simplify calls

* test: prevent crash on stopJob
2026-03-06 20:42:38 -05:00
Barış Soner Uşaklı
e1423636a5 feat: closes #13578, increase uniquevisitors
on ap pageviews like normal pageviews
2025-08-05 10:46:10 -04:00
Barış Soner Uşaklı
559a2d233d feat: add ap pageviews analytics 2025-07-11 15:09:55 -04:00
Barış Soner Uşaklı
669755d1e9 test: dont clear local when testing 2025-02-18 11:57:28 -05:00
Barış Soner Uşaklı
77dd6dd0e7 refactor: single remove 2025-02-14 10:42:05 -05:00
Barış Soner Uşaklı
d872470843 refactor: cleanup ip:recent 2025-02-14 10:25:59 -05:00
Julian Lam
b106a6a018 feat: added analytics.peek 2024-05-14 12:05:53 -04:00
Barış Soner Uşaklı
e45f513bfe fix: #11756, fix unique visitor stats in acp table 2023-06-26 09:03:29 -04:00
Julian Lam
9b753d6d57 TTL Cache (#10816)
* refactor: move src/cacheCreate.js to src/cache/lru.js

* fix: call new library location for lru cache creator

* feat: add ttl cache

* fix: update upload throttler to use ttl cache instead of lru cache

* chore: add missing dependency

* fix: avoid pubsub conflicts

* fix: use get instead of peek, which is not available in ttl-cache
2022-08-10 15:26:07 -04:00
Julian Lam
c07d595662 Update to lru-cache@^7 (#10815)
* chore(deps): bump lru-cache from 6.0.0 to 7.13.1 in /install

Bumps [lru-cache](https://github.com/isaacs/node-lru-cache) from 6.0.0 to 7.13.1.
- [Release notes](https://github.com/isaacs/node-lru-cache/releases)
- [Changelog](https://github.com/isaacs/node-lru-cache/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-lru-cache/compare/v6.0.0...v7.13.1)

---
updated-dependencies:
- dependency-name: lru-cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(lru-cache): remove unneeded `length` params for cache creation, as `maxSize` was not used in those init calls, also renamed some methods to match new method names in lru-cache [breaking]

Added deprecation notices for old params

* fix: replace three direct calls to lru-cache with call to cacheCreate, moved cache creation call in uploads to run on first init as config is not populated at lib init

* test: move configs init above cache reset calls in databasemock

* move some more code above cache clear

* refactor: remove unused

* test: lru

* test: more debug

* test: on more test

* use await helpers.uploadFile

* fix: tests remove logs

* fix: acp cache page

* fix: add in one more guard again cache instantiation with `length` prop but no `maxSize` prop

* fix(deps): bump markdown

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Barış Soner Uşaklı <barisusakli@gmail.com>
2022-08-10 13:24:16 -04:00
Barış Soner Uşaklı
398777633f fix: closes #10436, fix DST issue on acp dashboard 2022-04-07 14:14:28 -04:00
Barış Soner Uşaklı
35fea58a57 refactor: only write analytics data on nbb that has runJobs=true
prevents mongodb index error when lots of nodebbs try to write to the same key
2021-12-21 23:09:01 -05:00
Barış Soner Uşaklı
8fb89d761a fix: don't crash server if analytics fails to save
fixes
MongoBulkWriteError: E11000 duplicate key error collection:
 community.objects index: _key_1_value_-1 dup key: { _key: "analytics:pageviews", value: "1639807200000" }
2021-12-20 16:28:03 -05:00
Barış Soner Uşaklı
2bb0828af6 refactor: dont need local function 2021-11-30 20:27:16 -05:00
sadaszewski
a1c9a69cd3 incr by bulk. (#9952)
* incr by bulk.

* Update analytics.js

* Update analytics.js

* lint: fix

Co-authored-by: Barış Soner Uşaklı <baris@nodebb.org>
Co-authored-by: Barış Soner Uşaklı <barisusakli@gmail.com>
2021-11-30 20:26:07 -05:00
Julian Lam
5b42b6b369 API route for returning tracked analytics keys (#10019)
* feat: track metrics saved by NodeBB (and assoc. plugins), #9949

* feat: route to retrieve analytics keys, closes #9949
2021-11-19 15:12:13 -05:00
Peter Jaszkowiak
cc9d6fd08b chore: eslint max-len 2021-02-08 18:06:44 -05:00
Peter Jaszkowiak
5c2f0f0557 chore: eslint no-restricted-syntax 2021-02-08 18:06:44 -05:00
Peter Jaszkowiak
b56d9e12b5 chore: eslint prefer-arrow-callback 2021-02-08 18:06:44 -05:00
Peter Jaszkowiak
707b55b6a5 chore: eslint prefer-template 2021-02-08 18:06:44 -05:00
Barış Soner Uşaklı
b8cafefce2 fix: winston usages 2020-11-26 13:56:34 -05:00
Julian Lam
6e2da9966e refactor: move plugin hook methods to plugin.hooks.* 2020-11-20 16:32:39 -05:00
Barış Soner Uşaklı
a05905f196 performance improvements (#8795)
* perf: nconf/winston/render

cache nconf.get calls
modify middleware.pageView to call next earlier
don't call winston.verbose on every hook see https://github.com/winstonjs/winston/issues/1669
translate header/footer separately and cache results for guests

* fix: copy paste fail

* refactor: style and fire hook only log in dev mode

* fix: cache key, header changes based on template

* perf: change replace

* fix: add missing await

* perf: category

* perf: lodash clone

* perf: remove escapeRegexChars
2020-10-26 10:43:18 -04:00
psychobunny
14ba1a6dbb feat: analytics:maxCache setting in ACP
I removed a TODO from core. Watch your head as flying pigs are abound
2020-09-29 17:31:08 -04:00
Barış Soner Uşaklı
ce6b20c4b7 feat: remove topics.async.getTopicData usage 2020-07-24 12:02:16 -04:00
Barış Soner Uşaklı
e80379dc0e feat: display stack trace on winston.error 2020-06-20 23:32:12 -04:00
Barış Soner Uşaklı
dcb85ee7a1 #8344 (#8346)
* feat: wip

* feat: wrap middlewares

* feat: middleware errors

* feat: more middleware changes

* fix: remove unused async

* fix: prevent version errors from blocking acp render

* feat: wrap more middlewares
2020-06-03 20:18:42 -04:00
Julian Lam
398f0120b8 fix: logic for determining dailyStats hour vars
Reverts 9ecdb92f7c and fixes it
properly.
2020-02-07 11:35:45 -05:00
Julian Lam
9ecdb92f7c fix: fix daily analytics being one day off 2020-02-07 11:20:59 -05:00
Julian Lam
0b9ad416cb fix: remove debug line 2020-02-07 10:51:24 -05:00
Julian Lam
79737c53fd feat: guard against accidental ommision
...of analytics: prefix in get*StatsForSet methods
2020-02-07 10:28:40 -05:00
Barış Soner Uşaklı
223c108cc6 refactor: async/await analytics 2019-09-23 00:04:57 -04:00
Julian Lam
a0c0ef1ba4 feat: ACP analytics API route (#7725)
* feat: added API route for retrieving analytics via REST API

* feat: sets is now optional, can pass in multiple sets

* fix: moved expand and added json button to panel header

* fix: matching api params to socket method

* fix: update json api button url on graph change

* fix: updated default counts based on passed in units
2019-06-28 14:59:55 -04:00
Barış Soner Uşaklı
4d9bc30d1f feat: async3 upgrade (#7639)
* feat: async3 upgrade WIP

* fix: async.doWhilst

* fix: async early exit

* fix: psql doUntil

* fix: psql again
2019-05-30 14:36:33 -04:00
Barış Soner Uşaklı
29b63ae7fb fix: #6979 2018-12-14 17:31:06 -05:00
renovate[bot]
64b9dabff8 chore(deps): update dependency eslint-config-airbnb-base to v13 (#6599)
* chore(deps): update dependency eslint-config-airbnb-base to v13

* chore: #6599, linting 😬
2018-11-07 15:53:14 -05:00
Barış Soner Uşaklı
9f36ce3997 #6862 2018-10-24 11:24:37 -04:00
Dravere
956aa55253 Added server hook action:analytics.increment 2018-06-15 15:00:57 -04:00
Julian Lam
6ce635deb5 closes #6539 2018-06-01 15:15:57 -04:00
Barış Soner Uşaklı
d31d8fa806 move category pageview to controllers
so topic views are tracked as category views
2018-06-01 14:57:59 -04:00
Peter Jaszkowiak
e609e497b3 Provide more error information to logs
Provides full stack instead of just message
2017-11-01 18:59:48 -06:00
Julian Lam
53ca6d1199 closes #5889 2017-08-24 12:39:57 -04:00
Julian Lam
e4adaa8959 linting 2017-05-11 17:22:47 -04:00
Julian Lam
816809ce3c fixes #5606 2017-05-11 16:53:35 -04:00
Julian Lam
72e77772db Merge branch 'master' into develop 2017-02-24 12:46:40 -05:00
Peter Jaszkowiak
896c8c7343 ESlint object-curly-spacing 2017-02-18 12:30:49 -07:00
Peter Jaszkowiak
604358ecc4 ESlint keyword-spacing, no-multi-spaces 2017-02-18 01:52:56 -07:00
Peter Jaszkowiak
b1b486f729 ESlint quote-props 2017-02-18 01:19:20 -07:00