Commit Graph

507 Commits

Author SHA1 Message Date
Ben Lubar
97a3079bc0 Optimize PostgreSQL session store.
- Sets session ID to be a constant-sized character field inline in the
  table instead of a variable-length character field compressed outside
  of the row.
- Changes the session data from "json" to "jsonb" which is faster and
  smaller on disk with the drawback of not maintaining key order and
  whitespace (which we don't care about at all)
- Adds a clustered index on the session expiration timestamp, which will
  make periodic session purging much faster.

(For an example, WTDWTF currently has 482355 sessions in the table, and
 because there is no index on the expiration timestamp, PostgreSQL must
 check all the rows in the table every time.)

This upgrade script can run concurrently with a live NodeBB instance,
but any action that touches sessions will wait until the upgrade script
is done.

Does not touch the database if the database driver is not set to
PostgreSQL or if there is a Redis database configured.
2018-09-02 00:32:55 -04:00
Barış Soner Uşaklı
0519f84734 Promisify modules (#6723)
* WIP promisify

* promisify psql

* ability to skip some keys

* dont promisify client object

* remove async

* clone entire module so it has all properties

* add shim for node 6

* ignore sessionStore as well

* ignore pool on psql
2018-08-31 11:04:42 -04:00
Ben Lubar
33228bb7fe PostgreSQL database driver (#5861)
* [test/database/list] Fix test list 4 being used in two different tests

* [database/postgres] PostgreSQL database driver

* [database/postgres] Make transactions work based on continuation scope.

* [database/postgres] Implement nested transactions

* eslint --fix

* Add database changes from earlier this week to the PostgreSQL driver.

* Fix typo

* Fix postgres.incrObjectFieldBy returning undefined instead of null when given NaN

* [database/postgres] Fix sortedSetsCard returning an array of strings.

* Update socket.io postgres adapter

* Fix PostgreSQL erroring when multiple updates are made to the same sorted set entry in a single operation.

Add a test case to catch this error.

* Fix lint errors.

* Only prune sessions on one instance in a cluster to avoid deadlocks.

They're caught and handled by the database server, but they spam the logs.

* Fix arguments.slice.
2018-08-08 15:13:48 -04:00
Baris Usakli
16a7728ab8 fix redis warnings 2018-07-06 14:20:37 -04:00
Barış Soner Uşaklı
fe80186ddf add withscores 2018-06-22 16:28:22 -04:00
Barış Soner Uşaklı
8263f81d0c fix pubsub when only mongodb is used 2018-05-25 11:56:40 -04:00
Barış Soner Uşaklı
3b5d6951bb closes #6499 2018-05-10 16:31:16 -04:00
Barış Soner Uşaklı
77d47b31fb cleanly shutdown
wait for webserver to stop accepting connections
destroy current connections
wait for db connection to close
2018-03-22 16:36:23 -04:00
Baris Usakli
3c4b4f5263 don't call calback twice #6375 2018-03-21 12:25:25 -04:00
Barış Soner Uşaklı
be5c237388 fix decr/incrObjectField if key in array doesn't exist create it 2018-03-17 19:08:52 -04:00
Barış Soner Uşaklı
6a9a73c86c changes to leaveAllGroups
Groups.destroy can take an array of groupnames
Groups.leave can take an array of groupnames
db.incrObjectField/decrObjectField can take an array of keys
db.sortedSetRemove can take an array of keys and values
db.setRemove can take an array of keys
2018-03-17 18:49:38 -04:00
Baris Usakli
38acedf5bb lint 2018-03-15 12:18:54 -04:00
Baris Usakli
959a742c63 add llen to dbal 2018-03-15 12:06:40 -04:00
Barış Soner Uşaklı
bce640e514 closes #6340 2018-03-03 12:31:27 -05:00
Barış Soner Uşaklı
7a2d6fe22d Merge remote-tracking branch 'refs/remotes/origin/master' into develop 2018-02-02 10:36:25 -05:00
Barış Soner Uşaklı
b084a596fa closes #6301 2018-02-01 14:33:06 -05:00
Barış Soner Uşaklı
c0b7d4f5b5 fix redis export 2018-01-22 16:51:12 -05:00
Barış Soner Uşaklı
8a6345ded7 possible fix for pubsub 2018-01-22 16:41:22 -05:00
Ben Lubar
e85aabbe74 Allow running as a cluster without Redis (#6233)
* [database/*] Allow databases other than Redis to provide pubsub for clustering if Redis is not present

* [pubsub] Delay messages sent before the database is ready until the database is ready.

* [pubsub] Restore old behavior of not using the database in non-clustered NodeBB instances.

See comment: https://github.com/NodeBB/NodeBB/pull/6233#issuecomment-357814968
2018-01-18 13:02:56 -05:00
Julian Lam
457194b333 Merge branch 'master' into develop 2018-01-17 11:02:02 -05:00
Barış Soner Uşaklı
e092778ba6 closes #6200 2018-01-15 15:05:33 -05:00
Ben Lubar
5b1ed21634 [database/mongo] Improve speed of sortedSetRank (#6229)
* [database/mongo] Improve speed of sortedSetRank

* [database/mongo] Fix sortedSetRank to filter by _key
2018-01-12 13:32:07 -05:00
Barış Soner Uşaklı
5146f43b33 remove console.log 2018-01-03 14:38:09 -05:00
Barış Soner Uşaklı
87d40e9e78 fix increment to use data 2018-01-03 14:24:07 -05:00
Barış Soner Uşaklı
cbaa977233 fix type on mongodb if key has expireAt 2017-12-30 11:41:36 -05:00
Barış Soner Uşaklı
ea8cf6545c change db.set/get to use data field instead of value 2017-12-18 20:21:38 -05:00
Barış Soner Uşaklı
8e04e86e24 closes #5912 2017-12-07 20:43:07 -05:00
Baris Usakli
5ca3b535e7 closes #6105 2017-11-22 12:19:08 -05:00
Baris Usakli
dbf815ed47 closes #6103 2017-11-22 12:10:40 -05:00
Baris Usakli
30ae0a5499 another process.nextTick 2017-11-15 13:06:23 -05:00
Baris Usakli
e187a0def2 fix method name 2017-11-15 11:50:44 -05:00
Baris Usakli
12546cb9ad break up sync operation 2017-11-15 11:35:25 -05:00
Julian Lam
7e0d0e0eb5 Merge remote-tracking branch 'origin/develop' 2017-11-02 14:13:48 -04:00
Barış Soner Uşaklı
54a84bf130 cache fixes 2017-11-02 12:56:44 -04:00
Barış Soner Uşaklı
43f9c3224f dont crash sortedSetIncryBy on redis if no callback is supplied 2017-11-02 08:14:35 -04:00
Peter Jaszkowiak
e609e497b3 Provide more error information to logs
Provides full stack instead of just message
2017-11-01 18:59:48 -06:00
Barış Soner Uşaklı
a73c2628c4 cache improvements, stats 2017-10-25 18:20:44 -04:00
Barış Soner Uşaklı
e7318a1cea Mongo cache (#6002)
* cache mongodb objects

remove user settings cache since it is cached at the object level

* removed indexOf from isMemberOfSets

* change has to get, change maxAge to 0
2017-10-23 18:31:16 -04:00
Barış Soner Uşaklı
7ad4420b6a closes #6000 2017-10-23 15:09:13 -04:00
Barış Soner Uşaklı
108f84fd19 add type to dbal 2017-10-20 13:51:07 -04:00
Barış Soner Uşaklı
62c81ca0d3 fix lint 2017-08-20 21:07:54 -04:00
Barış Soner Uşaklı
329bfeeb3b convert min/max to float 2017-08-20 20:41:31 -04:00
Barış Soner Uşaklı
923bd86662 Widgets refactor (#5817)
* widgets refactor

render widgets server side
widgets can use all the data the template can use

* fix tests
2017-07-10 12:54:45 -04:00
Baris Usakli
e0de922ede Revert "widgets refactor"
This reverts commit 9e94edb5aa.
2017-07-07 15:35:06 -04:00
Baris Usakli
9e94edb5aa widgets refactor
render widgets server side
widgets can use all the data the template can use
2017-07-07 15:33:16 -04:00
Baris Usakli
d469f53eba using async in batch
removed unused code in debug
2017-07-06 12:08:01 -04:00
Barış Soner Uşaklı
3d85992757 modify db.processSortedSet so it works with intervals 2017-06-23 18:18:34 -04:00
Baris Usakli
7fe239229d show 3 digits 2017-06-20 16:34:34 -04:00
Peter Jaszkowiak
3c2d4fe649 Switch from underscore to lodash 2017-05-27 14:51:45 -06:00
Barış Soner Uşaklı
890c2eff70 style changes 2017-05-27 01:44:26 -04:00